IPRED validation now drops subjects without ETAs (like dose-only
subjects) from the input data however the $INPUT columns are ordered.
Before, this only happened when ID was the first column; otherwise the
solve got fewer ETA rows than subjects and the IPRED validation was
skipped. The ETA rows are matched to the individuals (contiguous runs of
an ID) in data order, so a reused ID stays a separate individual even
when a dropped subject between them leaves them next to each other, and a
TIME reset within an individual does not split it. The PRED validation keeps the
subjects the IPRED validation drops (#269).
MPAST(i) now translates to time > MTIME(i) instead of time >= MTIME(i). NONMEM evaluates the interval that ends at MTIME(i) with
MPAST(i) = 0; it becomes 1 only for later intervals. With rxode2's
continuous time both forms give the same solution, but with
rxSolve(nonmem = TRUE) the old form moved the change point one interval
early (#267).
When rxode2 supports rxSolve(nonmem = TRUE) (nlmixr2/rxode2#1429),
rxSolve() on a nonmem2rx model and the validation of an import solve
with nonmem = TRUE and addlKeepsCov = FALSE. Statements that do not
depend on a state ($PK) then read time as the time of the data record
that ends the interval, the way NONMEM calls $PK at its records. With
covsInterpolation = "nocb" this matches NONMEM for TIME in $PK,
MTIME change points and covariates changing between ADDL doses. With
older rxode2 versions the solve keeps addlKeepsCov = TRUE as before.
Models with delay() (the ADVAN16/ADVAN18 translations) keep the
continuous time, since rxode2 evaluates a delay() assigned outside of
d/dt() at the record time with nonmem = TRUE. The stress kit uses the
same options, so its time-in-pk case is no longer a known issue with a
newer rxode2. Its mtime-change-point case is no longer a known issue:
the ~1% difference between linCmt() and the ODE came from the kit's own
simulation switching at t >= MTIME (#267).
Added a NONMEM stress kit in inst/stress (like babelmixr2's). It
simulates edge-case datasets with rxode2, runs NONMEM, imports the runs
with nonmem2rx() and checks the import; without NONMEM it checks the
translations only. Load it with
source(system.file("stress", "stress.R", package = "nonmem2rx")) and
run stressCheck() and stressKit(); see inst/stress/README.md.
Runs with several $EST records (for example ITS followed by FOCE) now
import the final estimates. NONMEM writes one table per estimation step
to the .ext, .phi, .cov, .cor, .coi and .grd files and one
block per step to the .xml and .lst files, and nmext(), nmcov(),
nmgrd(), nmxml(), nmlst() and nminfo() read the first step. The
model was then validated with the first step's estimates against tables
from the last step. They now use the last estimation step of the first
problem.
First-order (METHOD=0) runs now validate with the POSTHOC etas from the
output tables. NONMEM leaves every .phi eta at zero for FO, and the
.phi etas were preferred.
nmxml() now reads $SIGMA (its XPath was malformed, so it was never
found), and returns NULL instead of an empty vector when $OMEGA is
missing.
ADVAN12 TRANS4 (CL V2 Q3 V3 Q4 V4 KA) models now translate to a working
linCmt() model. The parameter map was registered under TRANS2, so the
model kept an unresolved linCmtFun parameter and could not be solved. A
closed-form ADVAN/TRANS combination without a linCmt() translation is
now an error instead.
ADVAN4 TRANS6 (ALPHA BETA K32 KA) models now translate to linCmt().
A $MODEL that mixes named and bare COMP records (for example
COMP=(CENTRAL,DEFDOSE,DEFOBS) COMP) now renames the named compartments.
Previously the rename failed silently, and ADVAN5/ADVAN7 models then
stopped with an rxode2 syntax error because the matrix-exponential code
used the names the rename should have applied.
Abbreviated code now accepts the Fortran .NOT. operator and the /=
and .NEN. not-equal operators (previously a syntax error).
Model validation now matches the rxode2 solve to the NONMEM records by row,
so it is no longer skipped when the lengths differ: models with MTIME
(rxode2 adds output at the model event times) and data with
compartment-off (CMT<0) records now validate.
PRED validation now treats a reused, non-contiguous NONMEM ID as a new
individual (as IPRED validation already did). IPRED validation also keeps
ETAs aligned for reused IDs instead of skipping the check.
Validation no longer stops with "object '.iwres' not found" when the IPRED solve fails.
$OMEGA/$SIGMA BLOCK(n) records using the SD, CORRELATION or
CHOLESKY options now import the whole block. Previously only the first
row was kept (for example BLOCK(2) SD CORRELATION 0.3 0.5 0.2 became
eta1 ~ 0.09 and eta2 was lost). CHOLESKY values are now read as the
lower-triangular factor, and FIX is kept through the conversion.
Models that refer to DV (for example IRES = DV - IPRED in $ERROR)
no longer simulate wrong amounts when doses have modeled lag times and
durations in more than one compartment (#263). The DV reference makes
DV a time-varying covariate, which exposed a bug in rxode2; this
needs rxode2 5.1.8 or later.
$TABLE records now accept the VARCALC= and FIXEDETAS= options
instead of stopping with a syntax error (#262). They only change how
NONMEM computes extra table metrics, so they are parsed and ignored.
Unrecognised KEY=VALUE $TABLE options now also accept numeric and
quoted values, and their value (for example FOO=ETA1) is no longer
mistaken for a table column.
A $TABLE PARAFILE= value no longer replaces the table's FILE= name.
Fixed a heap-use-after-free reported by CRAN's ASAN/valgrind checks. The
record name used in syntax-error messages was stored in the parser's
duplicated-string pool, which is freed after every parse, so a later syntax
error (for example in a .lst file) printed freed memory. The record name
now has its own buffer.
An $OMEGA label left over from a translation that stopped with an error
(for example a NAMES() count mismatch) is now cleared before the next
$OMEGA record, instead of being read from freed memory.
Syntax errors in a .lst file are now labelled as coming from the NONMEM
output rather than from the last control-stream record parsed (or from a
NULL record name).
The PRED validation no longer needs the tibble package. It asked
rxSolve() for a tibble while the IPRED validation beside it asked for a
data.frame, so where tibble was unavailable the solve failed quietly and
only the IPRED half of $meta$validation was reported. The tibble also
failed, whatever was installed, for models that output a variable twice
(such as ADVAN5/ADVAN7 matExp() models).
A THETA(#) or ETA(#) used in the model beyond what $THETA or
$OMEGA defines is now an error. Previously it was silently translated
into a data covariate (like eta11), giving a model that could not be
used with the intended data.
An empty ; comment line no longer swallows the line after it. In
$THETA/$OMEGA this silently dropped an estimate and renumbered every
one after it (and likewise in the other record parsers).
Reading a NONMEM .lst covariance block is no longer slow. lst.g let a
run of numbers be cut into constant_lines in exponentially many ways, and
dparser resolves that ambiguity by greediness, which re-walks the whole
accumulated parse tree at every split point -- so the cost grew with the cube
of the block size (40 rows took 26 seconds). A statement is now a single
item rather than a line of them, which the tree walk never distinguished
anyway: inst/run-153.lst imports in 0.17s instead of 2.24s,
DDMODEL00000301/run3.lst in 0.15s instead of 1.57s, and 400 rows -- which
did not finish in any usable time before -- parse in 0.3s. A + still has
to lead an item, so a stray one is still a syntax error (#250).
$THETA and $OMEGA/$SIGMA records parse about 2.5x faster. Both
grammars had the ambiguity found in rxode2's tran.g: a nullable
statement under (statement)+, which lets an empty statement be inserted
at every position, and which dparser resolves by greediness. omega.g also
derived the empty string three different ways through block_type. The
emptiness moved to the list, so an empty record and $OMEGA BLOCK(n) SAME
(which carries no statements) still parse. A 400-theta $THETA record goes
from 7.3s to 2.8s (#250).
Both grammars now report no ambiguity at all under dparser's
ambiguity_fn. What is left of their super-linear cost is not ambiguity but
GLR branching from the optional ')' and ',' in theta.g's theta1..
theta8 and in omega.g's omega0..omega4, which is a separate matter.
The remaining inst/*.g grammars (abbrev.g, abbrec.g, data.g,
input.g, model.g, sub.g, tab.g) were measured and are flat in the
record size.
Validation plots now separate multiple endpoints (#171). When the
observations come from more than one DVID (or, without a DVID, more
than one CMT), the predCompare, ipredCompare and iwresCompare
datasets gain a final ENDPOINT column (like "DVID=2"). plot() and
autoplot() then facet the NONMEM vs rxode2 overview by type and
endpoint, and the per-subject pages draw one panel for each subject and
endpoint with its own y scale, so endpoints on different scales are no
longer drawn on the same axis. Single-endpoint output is unchanged.
$DATA now accepts every option in the NONMEM $DATA usage, and the ones
that change the data are applied when the input data is imported (#181):
TRANSLATE=(TIME/F[/D], II/F[/D]) divides TIME/II by F and rounds to
D digits (for example TIME/24 for hours to days, II/0.01/6). The
item may be named by its $INPUT synonym.TRANSLATE, NM-TRAN's day-time translation is done: hh:mm or
hh:mm:ss clock times and DATE/DAT1/DAT2/DAT3 dates (with
LAST20= for two digit years) become times relative to each
individual's first record, and hh:mm II values become hours.RECORDS=n (also NRECORDS, RECS, NRECS) is now applied before
IGNORE/ACCEPT, as NM-TRAN does, and RECORDS=label (ID, IR,
INDREC, ...) is supported.NULL=c replaces null data items and MISDAT=r values are read as 0.DROP/SKIP items are kept until after IGNORE/ACCEPT, so they can
be used in filter conditions.*, CHECKDATA, NOOPEN, BLANKOK, REPL,
(NO)FDATACSV, PRED_IGNORE_DATA, the /= operator, unquoted
character values (GEN.EQ.M compares with the string M, not a
column) and lower/mixed case options are now parsed.IGNORE/ACCEPT condition on a missing value is not met, so ACCEPT
drops such records; an IGNORE/ACCEPT list that cannot be parsed is
now a syntax error instead of being silently skipped.Importing a dataset with many reused NONMEM IDs is no longer slow. The
next free alias was found by walking every alias already given out and
scanning the whole level list for each, so the cost grew with the cube of
the number of reused blocks: 4000 blocks took about 10 seconds. Only the
next free suffix per id is remembered now, and the same 4000 blocks are
below timer resolution.
Importing a dataset where a NONMEM ID starts over more than twice no
longer hangs. A repeated ID is aliased <id>#2, <id>#3, and so on, but
the alias counter never advanced, so the third block of an id searched for
#2, found it, and searched for #2 again without end. Two blocks were
fine, which is why this went unnoticed; three or more -- an id reused after
a time reset, as in crossover and multiple-occasion data -- never returned.
nonmem2rx now requires rxode2 5.1.5 or later. That release fixes an
rxode2 model-piping bug where a model piped from an import (which keeps a
persistent meta environment) shared the original's cached simulation
model, so states appended by piping (like a running AUC) were silently
dropped from the solve (#246).
NONMEM delay differential equation (DDE) models are now translated to
rxode2's native delay()/past() syntax. The DDE solvers ADVAN16
(RADAR5) and ADVAN18 (DDE_SOLVER) are accepted (in addition to the
ADVAN13 DDE extension), a delayed state AD_x_y (state A(x) delayed by
TAUy) becomes delay(rxddta<x>, TAU<y>), and a past history
AP_x_y = expr becomes past(rxddta<x>, TAU<y>) <- expr (NONMEM T maps to
rxode2 t). A constant past that equals the compartment's initial condition
is dropped, since rxode2 uses the initial condition as the default delay
history. The translated models solve in rxode2 (dense auto-switch) and can
be estimated in nlmixr2. ADVAN17 (delay differential algebraic
equations) is not supported.
Regenerate the rxSolve.nonmem2rx() method so it no longer passes the
order solver argument, which was removed from rxode2::rxSolve().
Passing it caused solving a translated model to fail with
unused argument: 'order' against the current CRAN rxode2.
nonmem2rx()'s inputData argument now also accepts a data.frame
of the already read-in NONMEM input dataset (in addition to a file
path). This is useful when importing a model from a different
system where you have the data to validate against but the file
paths in the control stream do not match (#186). The columns are
assumed to be in $INPUT order and the usual $INPUT names,
DROP, IGNORE/ACCEPT filters and record subsetting are applied.
ADVAN5/ADVAN7 general linear models (which NONMEM itself solves with
matrix exponentials) are now translated to rxode2's native
matrix-exponential matExp() model (cmt() declarations plus
k_<from>_<to> rate constants) by default, instead of explicit d/dt()
ODEs. The previous ODE translation is retained and can be selected with
the new matexp=FALSE argument (or options(nonmem2rx.matexp=FALSE)).
Other model types are unaffected, and if the installed rxode2 does not
support matExp() the ODE translation is used with a warning.
NONMEM mixture models ($MIX) now translate to the native rxode2/nlmixr2
mixture support (mix()), replacing the previous rxord() simulation of
the sub-population. When the mixture probabilities are simple parameters
(e.g. P(1)=THETA(5)), the imperative MIXNUM/MIXEST branching in
$PK/$PRED is collapsed into readable mix() calls (e.g.
V <- mix(VCM, p1, VCF)), the probabilities are registered on the model
(ui$mixProbs) so the model estimates natively under focei and saem,
and MIXEST/MIXNUM map to the reserved mixest component. Models whose
probabilities are not simple parameters fall back to the previous rxord()
translation. Simulation-based validation remains gated for mixtures because
NONMEM's per-subject sub-population assignment is not recoverable for a
faithful prediction comparison.
Support the NONMEM $DATA numeric-comparison operators .EQN. and
.NEN. in IGNORE=/ACCEPT= filters (#195). These request that the
data item be converted to numeric before being compared, so they now
translate to as.numeric(.data$COL) == value / != value. .NEN.
was previously unparseable (silently dropping the filter), and .EQN.
was treated as a plain == without the numeric coercion.
Support assignment to the NONMEM COM(#) communication array (declared
with $ABBREVIATED COMRES=#). Previously only reading COM(#) was
translated, so control streams that assigned COM(#) (e.g.
IF(NEWIND.LE.1) COM(1)=-1) failed to parse (#228). A COM(#)
assignment now translates to the rxode2 sticky variable rxCOM_#_,
which retains its value from record to record like the NONMEM COM
array (see the rxode2 sticky variable vignette).
Add optional LLM-assisted residual error detection to as.nonmem2rx()
(via the ellmer package) for models imported without a residual
error specification ($predDf). When no chat engine is supplied,
the default engine is now chosen dynamically: getOption( "nonmem2rx.llmProvider") is honored first (accepting any exported
ellmer::chat_* engine by name), otherwise the first provider with a
configured API key is auto-detected (e.g. ANTHROPIC_API_KEY,
OPENAI_API_KEY, GEMINI_API_KEY), instead of always requiring
Claude. Any ellmer chat object may also be passed directly through
the new chat argument.
Add integer overflow guards in the C-level string buffer
(src/sbuf.c). sAppendN, sAppend, and addLine previously
computed the new allocation size as sbb->o + 2 + n + SBUF_MXBUF
(or analogous expression). When the user-controlled n was large
enough this expression overflowed int to a negative value, which
R_Realloc then converted to a huge unsigned size and crashed. The
guard converts this into a clean R error.
Document known (int)strlen(gBuf) cast in all 10 NONMEM-record parser
entry-points (src/abbrec.c, src/abbrev.c, src/data.c,
src/input.c, src/lst.c, src/model.c, src/omega.c, src/sub.c,
src/tab.c, src/theta.c). Inputs at or above INT_MAX bytes cause
silent length truncation passed to dparse(). A long-term fix will
switch each call site to udparse() once dparser-R ships that symbol
to CRAN.
Fix implicit ptrdiff_t to int truncation in rc_dup_str
(src/records.c). When the parser passes a string segment longer
than INT_MAX bytes (or a NUL-terminated string of that length),
the pointer difference / strlen result was silently cast to int,
truncating the length to a wrong (often negative) value. The new
guard rejects such inputs with an informative R error.
Bug fix for covariance matrices that span multiple FORTRAN output pages
Bug fix for multiple errors are imported using nonmem2rx()
Change NONMEM import for mixture models to prepend NM to MIXNUM and related import items so they will not interact with rxode2/nlmixr2's handling of mixture models.
Try to make sure all the values that can be numeric are numeric #208
Use qs2 since qs is being archived
Better handling of ytype
Added work-around for new Rstudio completion
$ERROR (ONLY OBS) supportif statements can be upper or lower case #196Add more flexible replacement of ETA and other values #188
Add ability to use nonmem2rx with a NONMEM control stream text input
Be more forgiving in the validation and remove IDs without
observations when solving the IPRED problem.
Binary linkage to dparser changed to structure only, meaning
nonmem2rx may not have to be updated if dparser is updated.
When reading NONMEM results from xml will try nm: prefixed tags
and non-nm: prefixed tags.
Omega and Sigma prior estimates are currently ignored (theta priors were already ignored)
Improve reading in theta values from the xml
Read all NONMEM files using latin1 encoding to allow single byte parser to work
When lines in the NONMEM input dataset start with # they are now
ignored.
When all IDs are zero, NONMEM assumes restarting time gives different IDs; this is now reflected in NONMEM translation of IDs.
With linCmt() parsing, expand the scope of conflicting parameters
that will be renamed with an import.
Added better parsing for ELSE where there is another IF on the
next line.
Prefixed conflicting VP with rxm. when linCmt() models to be
more accommodating when importing linear compartment models.
Added explicit requirement for rxode2 2.0.13
Added support of DADT(#) statements on the right side of the
equation, i.e. DADT(3) = DADT(1) + DADT(2) (#164)
Added support of ADVAN#, TRANS# (#161)
Added more NONMEM-specific solving options
Fixed security related format issues as requested by CRAN #167
Now omega, thetaMat, dfObs and dfSub are incorporated into
model function (by default). You can change this with the
nonmem2rx keep argument
Using the rxode2 2.0.13 makes sure that the solves for models
where the endpoint is not determined in the typical nlmixr2 style
will validate more often (due to a bug in solving in rxode2).
Added support for ADVAN5 and ADVAN7 models
Add parsing of accept/ignore characters for example IGNORE=(C='C')
(See Issue #140)
Add more robust reading of NONMEM information (and add source) in
nminfo() (See issue #142)
Since NONMEM does not protect divide by zeros by default, the
default for solveZero is changed to solveZero = TRUE for
nonmem2rx objects.
Fixed bug for renaming eta and theta when they are renamed so
that the ui$iniDf does not match the theta# or eta# (Issue
#153)
Turned off testing of the as.nonmem2rx example since it took too
much time (according to CRAN)
LTO, valgrind etc)NEWS.md file to track changes to the package.