Frequently Asked Questions

. Can I train a deep learning model using RSMTool? [#]

rsmtool supports a large number of learners but not deep learning algorithms. However, you can still use rsmeval to obtain a comprehensive evaluation report from any model.

. How can I customize rsmtool and rsmeval reports? [#]

The reports generated by rsmtool and rsmeval are designed to be fully customizable. You can:

Omit sections using the general_sections configuration option;
Change the order of the sections using the section_order option;
Create your own “custom” sections.

. I get the following error: No responses remaining after filtering out non-numeric feature values. No further analysis can be run. What happened? [#]

rsmtool is designed to work only with numeric features. Non-numeric values including missing values are filtered out. Some of the common reasons for this error are:

The human score column or one of the features contains only non-numeric values. You can either exclude this feature or convert it to one-hot encoding
You have features with missing values. The solution is to replace missing values with zeros. Note that this applies even if you use the .jsonlines format. We have an open issue to provide an option to automatically convert missing values to zeros.
You have a lot of missing feature values and none of the responses has numeric features for every single feature. Inspect the *_excluded_responses files in the output directory to see what responses have been excluded.

. Can I pass a set of learners to the model option in the configuration file or do I need a separate run for each leaner I want to try? [#]

rsmtool cannot train multiple models via a configuration file. If you wish to use multiple learners/models, you should use the RSMTool API instead of the command line.

. Can I compute predictions on new data using a model trained with rsmtool ? [#]

Yes! We have built rsmpredict to do exactly this!

. Why did rsmtool change the sign of some features in feature.csv ? I thought rsmtool assumes a positive sign for all features? [#]

rsmtool indeed assumes a default positive sign for all raw features. However, if you set select_transformations to true in your experiment configuration file, RSMTool will automatically apply transformations to some of the features. Some transformations such as inv (inverse transform) change the polarity of the feature. In such cases, RSMTool takes this into account and changes the sign accordingly. See note here.

. Why does rsmpredict exclude a different number of responses compared to rsmtool ? [#]

This mismatch may happen due to the following:

First, rsmtool is run with one of the built-in feature selection models and excludes responses where any of the candidate features has missing/non-numeric values in either the training or the evaluation set. Next, it performs the feature selection, trains a model with those features using the filtered training set and generates predictions on the filtered test set.
Next, rsmpredict is run to generate predictions on the evaluation set using the trained model and excludes only those responses where any of the selected features (a subset of the original features) has missing/non-numeric values.

As a result, there is a mismatch in the number of predictions generated by rsmtool and rsmpredict.

. The relative betas for the linear regression model in my rsmtool run do not sum to 1. Is that a problem? [#]

Please check if your model has any negative coefficients. Relative coefficients only make sense when all model coefficients are positive. Their sum is expected to be less than 1 if there are negative coefficients. Note that if this is the case, the relative coefficients will not be included into the report.

. Can I use Weights & Biases to log artifacts and reports? [#]

Yes! W&B logging is natively supported by the following command-line tools: rsmtool, rsmeval, rsmxval, rsmexplain, rsmcompare, and rsmsummarize. To enable it, set the use_wandb configuration option to true and set the wandb_entity and wandb_project options appropriately.

. It’s so difficult to remember all the configuration options. [#]

We agree that it can be quite overwhelming to remember all the configuration options. This is why all command-line tools support automatic generation of configuration files!