The first place this message may occur is during calculation of the inverse of the numerator relationship matrix (I have never seen this happen though); if it does, you are likely to have a very big and very dense pedigree and would be best advised to reduce the size of your analysis.
The second possibility is that you are trying to use the PX-EM algorithm (this is the default for the first few iterates when there are more than 18 parameters to be estimated, so you may be doing this without having explicitly chosen to do so), and WOMBAT stops before the first iterate. WOMBAT has a guess on how many non-zero off-diagonal elements there are in one triangle of the coefficient matrix of the mixed model equations and then attempts to set this up and to determine the actual number. If this is too small, it will stop. A particular scenario when this has been found to occur it the case of an analysis trying to estimate a covariance matrix of reduced rank (option PC), where the rank is substantially less than the dimension of the matrix.
To get around this, you could try to:
use the AI-REML algorithm from the start (see run time option —-aireml), especially if you have good starting values
set the initial guess manually using the run time option —-choozhz
The third place this can happen is when using the iterative solutions module; again, setting the maximum number manually via the —-choozhz option may help.