The artifact download
The way the tool operates is described in Paper #253: Predicting Program Properties from "Big Code".
This tool is copyrighted by ETH Zurich.
The authors will be grateful if the contents of this artifact is not distributed for purposes other than artifact evaluation. The authors would like to be able to open source the tool themselves and/or to incorporate it into commercial software.
The artifact does not require a VM, as it only depends on Java 7 and bash scripts. The provided binary compiler.jar is a modified Google Closure Compiler binary.
The tool is also available for online use at http://jsnice.org/. The online tool uses Google Analytics, but it has hundreds of daily users so it is unlikely reviewers can be identified based on this, and the authors promise they will not attempt to.
One can of course infer names and types locally with the provided binary and models using the following instructions:
cp model-reg/nameemsvm* . cp model-reg/typeemsvm* .
java -jar compiler.jar --compilation_level=JSNICE --jsnice_infer=NAMES --jsnice_features=ASTREL,NODEFLAG,ARGALIAS,FNAMES <input-file>.js
java -jar compiler.jar --compilation_level=JSNICE --jsnice_infer=TYPES --jsnice_features=TYPEREL,TYPEALIAS,TYPERELALIAS <input-file>.js
The tool will take a few seconds at start-up to load the model, but the actual inference is typically fast for reasonably big input files.
Note: the http://jsnice.org/ website uses the same model trained of the full dataset like the ones included in this artifact, but trained with slightly different regularization constants. Thus, it may not always produce exactly the same names and types.
For each of our training data sizes, we include an eval1.sh script that evaluates the accuracy on the given dataset. For example, to evaluate the accuracy on the models trained on the full data, execute:
cd model-reg ./eval1.sh
The above command runs a tiny 4-line script that runs two evaluation commands - one for names and one for types. Each evaluation uses all the CPU cores of the machine and should run around 10 minutes on a modern 4-core CPU. As a result, evaluation reports for names and types are created (evalnames.txt and evaltypes.txt). To see a summary, look the last few lines of these files (e.g. via tail command). E.g. in evalnames.txt, there should be a TOTAL line that says:
>>>> Errors: 56. Names: OK=269180. DIFF=155222 (0.634257). Types: OK= 26. NO=2309. DIFF= 12 (p=0.684211 r=0.016191) -- TOTAL
and in evaltypes.txt:
>>>> Errors: 56. Names: OK=107254. DIFF=317148 (0.252718). Types: OK=1281. NO= 777. DIFF= 289 (p=0.815924 r=0.668939) -- TOTAL
The interpretation of these lines is, Error AST nodes due to minification (due to rearrangement of boolean conditions by uglifyjs, we ignore these, but they are negligibly few compared to the total number of names i.e. 56 vs >400K), and then for names:
After running an eval1.sh command, the directory evalresults is populated with the inferred types (names can also be obtained by changing eval1.sh not to evaluate types, which overwrites the evalresults directory).
Additional summary about the errors in an evaluation is produced in the type_errors.txt files. These files list the number of cases when JSNice predicted a type correctly or incorrectly:
To evaluate quality when structure would not be used (i.e. no relationship between inferred labels), run: ./eval_nostructure.sh and see the results in evalnames_nostruct.txt and evaltypes_nostruct.txt .
The runtimes for each evaluation run are at the end of the evalnames.txt and evaltypes.txt. E.g. a line:
Took on average 22.9660014781966msFor the full training data model (in model-reg), we also include similar results with different beam sizes and no beam (naive). These are in the files evalnames_<beamsize> and evaltypes_<beamsize>. To regenerate these files on your machine (takes a lot of time, especially for the naive), run:
cd model-reg ./eval_beam.sh ./eval_naive.sh
cd model-reg ./eval1.sh
Then, we perform type checking (from Closure Compiler) by calling:
./typecheck_eval.sh | tee typecheck_rate.txt
After this command, typecheck_rate.txt contains for each evaluated file (with existing type annotations, the number of type errors for the original code versus the number of type errors for the types inferred by JSNice). The values "<num> ERR" include only the errors we consider in the paper (excluding the inconsistent property check).
To only count the number of files that typecheck, in order to get numbers for Figure 6, grep through the typecheck_rate.txt file:
$ wc -l typecheck_rate.txt 396 typecheck_rate.txt $ grep ' 0 ERR vs' typecheck_rate.txt | wc -l 107 $ grep ' 0 ERR$' typecheck_rate.txt | wc -l 227
To run training, put your training data in a directory named "data" (where compiler.jar stays) and run:
java -jar compiler.jar --train_jsnice --jsnice_infer=NAMES --jsnice_features=FNAMES,ASTREL,NODEFLAG,ARGALIAS --jsnice_training_flags=SVM,MIDAPR
After the training, you can look at the best_features.txt or predict names for new programs
Note: JSNice will not train of files it detects as minified. Do not put all your statements one on line or use only one-letter parameter names.
In addition to model for names, model for types can be trained by running:
java -jar compiler.jar --train_jsnice --jsnice_infer=TYPES --jsnice_features=TYPEREL,TYPEALIAS,TYPERELALIAS --jsnice_training_flags=SVM,MIDAPR
The trained model can be used for prediction as described in the "Using the tool" section above
Via the parameter --jsnice_features, different features can be used to train a model and at prediction time.
All options exposed by Closure Compler plus the options added by JSNice are listed by running
java -jar compiler.jar --help