Lerna Mono Repos with Internal Dependencies

Mono repos are very popular nowadays for bundling a collection of related JavaScript libraries in one single repository. Figuring out how to handle dependencies between multiple packages inside that repository was a little tricky for me. This is how I got it working.

Why Lerna?

Outside of work, I first tried out the mono repo manager lerna with esgettext. Esgettext is written in TypeScript and - at the time of this writing - consists of two packages @esgettext/esgettext-runtime and @esgettext/esgettext-tools. The latter depends on the former because it is internationalized with @esgettext/esgettext-runtime.

Developing the two packages in parallel is cumbersome because you have to publish one package to your npm registry before you can use it from the other. Alternatively, you can use a git URL as a dependency but you then have to manually undo that before publishing.

With lerna, this problem is solved elegantly. In a nutshell, it does not download the internal dependency into the (sub-)package's node_modules but instead creates a symbolic link. But this is not sufficient for TypeScript and Jest. They require additional configuration.

Creating a Lerna Mono Repo

Instead of just looking at the code of esgettext we rather create a minimalistic mono repo with lerna from scratch.

If you are familiar with lerna, Jest, and Typescript, you probably want to skip the manual steps below and jump directly to Cloning the Current State.

But if you want to understand the individual steps, follow the instructions below. First we create an empty repository and initialize the structure for lerna:

$ mkdir lerna-deps
$ git init lerna-deps
Initialized empty Git repository in /path/to/lerna-deps/.git/
$ cd lerna-deps
$ npm init -y
Wrote to ...
...
$ npm add --save-dev lerna
...
$ npx lerna init
lerna notice cli v3.22.1
lerna info Updating package.json
lerna info Creating lerna.json
lerna info Creating packages directory
lerna success Initialized Lerna files

Our goal is to create a library that offers a function fortyTwo() that returns the number 42. A command-line tool fortyTwo will use that library to print the number 42 on the command-line.

Create a Lerna-Managed Package

Lerna sub-packages are created with the command lerna create in the directory packages. Let's start with the library:

$ cd lerna-deps
$ npx lerna create @forty-two/forty-two-runtime
... hit ENTER all the time
Is this OK? (yes) 
lerna success create New package @forty-two/forty-two-runtime created at ./packages/forty-two-runtime

The command has created some files and directories that we don't want. Let's delete them.

$ cd lerna-deps/packages/forty-two-runtime
$ rm -r README.md __tests__ lib

All that should remain is packages/forty-two-runtime/package.json.

You may have noticed that we have created a scoped package @forty-two/forty-two-runtime instead of just forty-two-runtime. This is very common for lerna-managed mono repos.

We also add two scripts to the top-level (!) package.json:

...
"scripts": {
    "bootstrap": "lerna bootstrap",
    "test": "lerna run test --stream"
},
...

Use TypeScript

We want to use TypeScript instead of vanilla JavaScript. Create a file lerna-deps/tsconfig.json, i. e. the top-level Typescript configuration:

{
    "compilerOptions": {
        "allowSyntheticDefaultImports": true,
        "noImplicitAny": true,
        "noImplicitReturns": true,
        "noImplicitThis": true,
        "noImplicitUseStrict": true,
        "removeComments": true,
        "declaration": true,
        "target": "es5",
        "lib": ["es2015", "dom"],
        "module": "commonjs",
        "sourceMap": true,
        "typeRoots": ["node_modules/@types"],
        "esModuleInterop": true,
        "moduleResolution": "node"
    },
    "exclude": [
        "node_modules",
        "**/*.spec..ts"
    ]
}

Your mileage may vary but this configuration should work.

Add typescript as a development dependency to your project:

$ cd lerna-deps
$ npm install --save-dev typescript
...
$

Use Jest

We want to use Jest for testing:

$ cd lerna-deps
$ npm install --save-dev jest ts-jest
...
$

We also need ts-jest for using Jest with Typescript. Add a top-level key "jest" to lerna-deps/packages/forty-two-runtime/package.json:

    ...
    "jest": {
            "moduleFileExtensions": [
                "js",
                "json",
                "ts"
            ],
            "rootDir": "src",
            "testRegex": ".spec.ts$",
            "transform": {
                "^.+\\.ts$": "ts-jest"
            },
            "coverageDirectory": "../coverage",
            "testEnvironment": "node"
        },
    ...

Now it's time to bootstrap the mono repo with the command lerna bootstrap. Since we have created a script for that command in package.json we can do:

$ cd lerna-deps
$ npm run bootstrap

> lerna-deps@1.0.0 bootstrap /path/to/javascript/lerna-deps
> lerna bootstrap

lerna notice cli v3.22.1
lerna info Bootstrapping 1 package
lerna info Symlinking packages and binaries
lerna success Bootstrapped 1 package

You should always run the bootstrap step after you have modified the mono repo structure.

Write a Test

First, change the script test in lerna-deps/packages/forty-two-runtime/package.json to look like this:

...
    "script": {
        "test": "jest"
    }

Now create the directory lerna-deps/packages/forty-two-runtime/src and a test file lerna-deps/packages/forty-two-runtime/src/forty-two.spec.ts inside of it:

import { FortyTwo } from './forty-two';

describe('forty-twp', () => {
    it('should produce forty-two', () => {
        expect(FortyTwo.magic()).toEqual(42);
    });
});

Now go back to the top-level directory and run the tests:

$ cd lerna-deps
$ npm test

> lerna-deps@1.0.0 test /path/to/lerna-deps
> lerna run test --stream
...
npm ERR! Test failed.  See above for more details.
$

The test failed as expected because the implementation is missing. Fix that by creating lerna-deps/packages/forty-two-runtime/src/forty-two.ts:

export class FortyTwo {
    public static magic() {
        return 42;
    }
}

Run the test suite again, and it should succeed:

$ cd lerna-deps
$ npm test

> lerna-deps@1.0.0 test /path/to/lerna-deps
> lerna run test --stream
...
lerna success - @forty-two/forty-two-runtime
$

If that does not work for you, execute npm run bootstrap again in the top-level directory.

Cloning the Current State

You can also copy the current state to your local machine with this command:

$ git clone https://github.com/gflohr/lerna-deps.git
...
$ cd lerna-deps
$ git checkout starter
$ npm install
...
$ npm run bootstrap
...
$ npm test
...
lerna success - @forty-two/forty-two-runtime
$

The tag "starter" contains this stage of the repository.

Create an Index File

By convention, the entry point of a TypeScript library should be a file index.ts. Create packages/forty-two-runtime/src/index.ts like this:

export * from './forty-two';

Creating a Tools Package

Now create the command-line interface to the runtime library in another sub-package with lerna create:

$ cd lerna-deps
$ npx lerna create @forty-two/forty-two-tools

Again, delete everything inside lerna-deps/packages/forty-two-tools except for package.json:

$ cd lerna-deps/packages/forty-two-tools
$ rm -r README.md __tests__ lib
$ mkdir src

Of course, we also want to test the command-line interface. We have to change the test script in lerna-deps/packages/forty-two-tools/package.json:

    "script": {
        "test": "jest
    }

And in the same file prepare Jest for using TypeScript:

    ...
    "jest": {
            "moduleFileExtensions": [
                "js",
                "json",
                "ts"
            ],
            "rootDir": "src",
            "testRegex": ".spec.ts$",
            "transform": {
                "^.+\\.ts$": "ts-jest"
            },
            "coverageDirectory": "../coverage",
            "testEnvironment": "node"
        },
    ...

Unit Test for Tool

Continue with writing a test in the directory lerna-deps/packages/forty-two-tools/src:

$ mkdir lerna-deps/packages/forty-two-tools/src

The test lerna-deps/packages/forty-two-tools/src/forty-two-cli.spec.ts should look like this:

import { FortyTwoCLI } from './forty-two-cli';

describe('forty-two cli', () => {
    it('should return 42 from the CLI wrapper', () => {
        expect(FortyTwoCLI.magic()).toBe(42);
    });
});

The test will fail because the implementation lerna-deps/packages/forty-two-tools/src/forty-two-cli.ts is still missing. Add it:

import { FortyTwo } from '@forty-two/forty-two-runtime';

export class FortyTwoCLI {
    public static magic() {
        return FortyTwo.magic();
    }
}

But running npm test in the top-level directory still fails. The error message is:

src/forty-two-cli.spec.ts:1:29 - error TS2307: Cannot find module './forty-two-cli' or its corresponding type declarations.

Making TypeScript Resolve Internal Dependencies

The first step needed to fix the problem is to tell TypeScript how to resolve the internal dependency. Create a file lerna-deps/packages/forty-two-tools/tconfig.json with this contents:

{
    "extends": "../../tsconfig.json",
    "compilerOptions": {
        "baseUrl": ".",
        "paths": {
            "@forty-two/forty-two-runtime": ["../forty-two-runtime/src"]
        }
    },
    "includes": ["./src"]

The important things are the two compiler options "baseUrl" and "paths".

The object "paths" maps imports to a seach list in the file system. Note that the "paths" option will not work without setting the "baseUrl"!

Making Jest Resolve Internal Dependencies

Modifying tsconfig.json is enough for making TypeScript resolve the internal dependency. But we also have to tell Jest how to do that. Open lerna-deps/packages/forty-two-tools/package.json again, and change the top-level object "jest" like this:

...
    "jest": {
            "moduleFileExtensions": [
                    "js",
                    "json",
                    "ts"
            ],
            "moduleNameMapper": {
                    "^@forty-two/forty-two-runtime$": "<rootDir>/../../forty-two-runtime/src"
            },
            "rootDir": "src",
            "testRegex": ".spec.ts$",
            "transform": {
                    "^.+\\.ts$": "ts-jest"
            },
            "coverageDirectory": "../coverage",
            "testEnvironment": "node"
    },
...

The new key is moduleNameMapper. It is an object that maps module names as regular expressions to paths in the file system.

Run npm test in the top-level directory again, and it will now succeed.

Adding the Dependency With Lerna

Check the contents of lerna-deps/packages/forty-two-tools/package.json. It doesn't have any dependencies. Trying to install @forty-two/forty-two-tools from an npm registry would therefore fail.

You can fix this problem with lerna add:

$ cd lerna-deps
$ npx lerna add @forty-two/forty-two-runtime

The important thing happening is that it will add the dependency @forty-two/forty-two-runtime to the package.json of @forty-two/forty-two-tools. What it also does is to populate the directory node_modules of the tools package and resolve the dependency there with a symbolic link so that npm or yarn will not attempt to download the package from the npm registry:

$ cd lerna-deps
$ ls -l packages/forty-two-tools/node_modules/@forty-two
total 0
lrwxr-xr-x  1 guido  staff  26 Sep 11 09:36 forty-two-runtime -> ../../../forty-two-runtime

That symbolic link will only exist in your local development environment. People that install the package from an npm registry like https://npmjs.com will just regularly download the dependency.

You can download the complete sources for this example from github:

$ git clone git@github.com:gflohr/lerna-deps

Or if you had previously checked out the intermediate state:

$ cd lerna-deps
$ git checkout master
$ git pull

Things That Are Missing

This is neither a TypeScript nor a comprehensive lerna tutorial. For example, the build step is completely missing. And the forty-two-tools package does not contain any real command-line script.

If you are interested in a complete example, have a look at the mono repo https://github.com/gflohr/esgettext. It uses the same methods as described here but adds all the missing parts. Furthermore, it shows how to make the runtime part work both in the browser and on the server with NodeJS.