LB4: Model discovery to create model file

[DivCorleone]

Feature proposal

Model discovery to create model file

Current Behavior

Model discovery do not create model file

Expected Behavior

I have a mySQL database already created, i would like to discover the models and create their model file.

versions:

@loopback/cli version: 1.0.1

@loopback/* dependencies:
  - @loopback/authentication: ^1.0.1
  - @loopback/boot: ^1.0.1
  - @loopback/build: ^1.0.0
  - @loopback/context: ^1.0.0
  - @loopback/core: ^1.0.0
  - @loopback/metadata: ^1.0.0
  - @loopback/openapi-spec-builder: ^1.0.0
  - @loopback/openapi-v3-types: ^1.0.0
  - @loopback/openapi-v3: ^1.0.1
  - @loopback/repository-json-schema: ^1.0.1
  - @loopback/repository: ^1.0.1
  - @loopback/rest: ^1.0.1
  - @loopback/testlab: ^1.0.0
  - @loopback/docs: ^1.0.1
  - @loopback/example-hello-world: ^1.0.1
  - @loopback/example-log-extension: ^1.0.1
  - @loopback/example-rpc-server: ^1.0.0
  - @loopback/example-todo: ^1.0.1
  - @loopback/example-soap-calculator: ^1.0.1
  - @loopback/service-proxy: ^1.0.0
  - @loopback/http-caching-proxy: ^1.0.0
  - @loopback/http-server: ^1.0.0
  - @loopback/example-todo-list: ^1.0.1
  - @loopback/dist-util: ^0.4.0

Using this connector loopback-connector-mysql i can connect to the database and retrieve models:

let dataSrc = new juggler.DataSource(name, {...});

dataSrc.on('connected', () => {
  dataSrc.discoverAndBuildModels(modelName, {relations: true}, (err, model) => {
    console.log('model:', model);
  })
})

The model is correctly imported, but the corresponding file is not created. I have found nothing in the doc about creating the file so i made a little somewhat in nodejs to generate the files:

var util = require('util'),
  process = require('child_process'),
  fs = require('fs'),
  path = require('path'),
  appDir = path.dirname(require.main.filename);

export function writeModel(name, schema, upperCaseName) {
  let stream = fs.createWriteStream(appDir + "/src/models/" + name + ".model.ts");

  let header = "import {Entity, model, property} from '@loopback/repository';\n\n"
             + "@model()\n"
             + "export class " + upperCaseName + " extends Entity {";

  let footer = "\n\tconstructor(data?: Partial<" + upperCaseName + ">) {\n"
             + "\t\tsuper(data);\n"
             + "\t}\n"
             + "}"

  let mapping = {};
  mapping['date'] = 'string';
  mapping['binary'] = 'boolean';

  stream.once('open', function(fd) {
    stream.write(header);

    for (let prop in schema) {
      console.log('prop: ', prop);

      let property = "\n\t@property({\n"
                   + "\t\ttype: '" + schema[prop].type.toLowerCase() + "',\n"
                   + (schema[prop].id ? "\t\tid: true,\n" : "")
                   + (schema[prop].require ? "\t\trequired: true,\n" : "")
                   + "\t})\n"

      let declaration = "\t" + prop
                      + (schema[prop].require ? "": "?") + ": "
                      + (mapping[schema[prop].type.toLowerCase()]
                        ? mapping[schema[prop].type.toLowerCase()]
                        : schema[prop].type.toLowerCase())
                      + ";\n";

     stream.write(property + declaration);
    }

    stream.write(footer);

    stream.end();
  });
}

The files are created, compiled by typescript, but when using lb4 repository and selecting a model, it has an error: Generation is aborted: Error: You did not select a valid model.

Example of generated file:

import {Entity, model, property} from '@loopback/repository';

@model()
export class Webmessagessent extends Entity {
	@property({
		type: 'number',
		id: true,
	})
	id?: number;

	@property({
		type: 'number',
	})
	messageid?: number;

	@property({
		type: 'number',
	})
	destid?: number;

	@property({
		type: 'date',
	})
	datetimecreation?: string;

	@property({
		type: 'date',
	})
	datetimeopen?: string;

	@property({
		type: 'date',
	})
	datetimetrash?: string;

	@property({
		type: 'binary',
	})
	visible?: boolean;

	@property({
		type: 'number',
	})
	reptomsgid?: number;

	@property({
		type: 'number',
	})
	destcompanyid?: number;

	constructor(data?: Partial<Webmessagessent>) {
		super(data);
	}
}

Is creating a model file from model discovery already a feature ?

Can we create a model file programmatically like the command line client lb4 would do it ?

Am i missing something when generating the file ?

See Reporting Issues for more tips on writing good issues

[dhmlau]

@DivCorleone , sorry that I may not be able to try your code snippet yet. By looking at the error Generation is aborted: Error: You did not select a valid model, it looks similar to the following issue.
loopbackio/loopback-connector-oracle#168 (comment)
and some more detailed screen caps in here: loopbackio/loopback-connector-oracle#168 (comment).
Hope to get you going first.

[dhmlau]

@DivCorleone, If you were able to select a model from lb4 repository command, it looks like your model is good to go. Please review my comment in loopbackio/loopback-connector-oracle#168 (comment) to make sure you have selected a model.

For model discovery, I found a way to somehow do it but I think there are rooms for improvements in terms of user experience. Perhaps @raymondfeng @bajtos or @marioestradarosa have a better way. Talking to @bajtos before, he was thinking to have a lb4 discover command for model discovery.

Let me share what I've done:

1. Discover the model
   Inspired by your code above, I got the model json printed out.
   If you create the datasource first using lb4 datasource, you don't need to specify the connection information.

let dataSrc = new juggler.DataSource('ds', {
      name: 'ds',
      connector: 'mysql',
      url: 'mysql://root:pass@localhost/testdb',
    });

    dataSrc.on('connected', async () => {
      const customerSchemas = await dataSrc.discoverSchemas(
        'Customer',
        options,
      );
      console.log(JSON.stringify(customerSchemas, null, 2));
    });

2. Copy the output from step 1 to a json file, say model.config.json.
   My model, Customer, has id and name as properties.

{
    "name": "Customer",
    "options": {
        "idInjection": false,
        "mysql": {
            "schema": "testdb",
            "table": "Customer"
        }
    },
    "properties": {
        "id": {
            "type": "Number",
            "required": true,
            "length": null,
            "precision": 10,
            "scale": 0,
            "id": 1,
            "mysql": {
                "columnName": "id",
                "dataType": "int",
                "dataLength": null,
                "dataPrecision": 10,
                "dataScale": 0,
                "nullable": "N"
            }
        },
        "name": {
            "type": "String",
            "required": true,
            "length": 200,
            "precision": null,
            "scale": null,
            "mysql": {
                "columnName": "name",
                "dataType": "varchar",
                "dataLength": 200,
                "dataPrecision": null,
                "dataScale": null,
                "nullable": "N"
            }
        }
    }
}

3. Within a LB4 app, run lb4 model --config model.config.json --yes
   The --config option allows you to feed in a file for creating model.
   The --yes option allows you to skip the default prompt.
   The output customer.model.ts is as below.

import {Entity, model, property} from '@loopback/repository';

@model()
export class Customer extends Entity {
  @property({
    type: 'Number',
    required: true,
    length: ,
    precision: 10,
    scale: 0,
    id: 1,
    mysql: [object Object],
  })
  id: Number;

  @property({
    type: 'String',
    required: true,
    length: 200,
    precision: ,
    scale: ,
    mysql: [object Object],
  })
  name: String;

  constructor(data?: Partial<Customer>) {
    super(data);
  }
}

Note: There are a few things that are valid, e.g. the type is upper case.

Hope it helps.

[marioestradarosa]

You don't need to generate the model using lb4 model if you don't want to. Once you get the model in json format as in step 2, you can use that in a model file as follows:

import {Entity, model, property} from '@loopback/repository';
@model(
//here paste te json file you copied on step 1 
)
export class Pluto extends Entity {}

As a separate note, I believe the lb4 model should support specifying this discovery generation automatically, probably lb4 model --table mysqlTable --datasource myDS .

So the final code would look (ie: pluto.model.ts) and then you can run lb4 repository, note that this will ask you for the property ID name, since it won't be able to infer it from the json model, since it doesn't have the id:true property, but you should be fine answering that to the wizard.

import {Entity, model, property} from '@loopback/repository';
@model(
{
    "name": "Customer",
    "options": {
        "idInjection": false,
        "mysql": {
            "schema": "testdb",
            "table": "Customer"
        }
    },
    "properties": {
        "id": {
            "type": "Number",
            "required": true,
            "length": null,
            "precision": 10,
            "scale": 0,
            "id": 1,
            "mysql": {
                "columnName": "id",
                "dataType": "int",
                "dataLength": null,
                "dataPrecision": 10,
"dataScale": 0,
                "nullable": "N"
            }
        },
        "name": {
            "type": "String",
            "required": true,
            "length": 200,
            "precision": null,
            "scale": null,
            "mysql": {
                "columnName": "name",
                "dataType": "varchar",
                "dataLength": 200,
                "dataPrecision": null,
                "dataScale": null,
                "nullable": "N"
            }
        }
    }
}
)
export class Pluto extends Entity {}

[dhmlau]

Thanks @marioestradarosa. I've created a task in capturing the documentation task and the potential enhancement.

[DivCorleone]

Thank you for your answers !
Effectively @dhmlau i didn't correctly select the model through lb4 repository (rookie mistake!).
With both you and @marioestradarosa's answer i came with :

export function writeModel(name, schema, upperCaseName) {
  let stream = fs.createWriteStream(appDir + "/src/models/" + name + ".model.ts");

  let header = "import {Entity, model, property} from '@loopback/repository';\n\n"
             + "@model(\n"

  let footer = "\n)\n"
             + "export class " + upperCaseName + " extends Entity {}";


  stream.once('ready', () => {
    stream.write(header);

    stream.write(JSON.stringify(schema, null, 2));

    stream.write(footer);

    stream.end();
  });

  stream.once('close', () => {
    fs.writeFileSync(appDir + "/src/models/index.ts"
      , "export * from './" + name + ".model';\n"
      , {flag : 'a'}
    );
  });

Creating a model file like:

import {Entity, model, property} from '@loopback/repository';

@model(
{
  "name": "Webmessagetypes",
  "options": {
    "idInjection": false,
    "mysql": {
      "schema": "...",
      "table": "webmessagetypes"
    },
    "relations": {}
  },
  "properties": {
    "id": {
      "type": "Number",
      "required": true,
      "length": null,
      "precision": 10,
      "scale": 0,
      "id": 1,
      "mysql": {
        "columnName": "id",
        "dataType": "int",
        "dataLength": null,
        "dataPrecision": 10,
        "dataScale": 0,
        "nullable": "N"
      }
    },
    "name": {
      "type": "String",
      "required": false,
      "length": 50,
      "precision": null,
      "scale": null,
      "mysql": {
        "columnName": "name",
        "dataType": "varchar",
        "dataLength": 50,
        "dataPrecision": null,
        "dataScale": null,
        "nullable": "Y"
      }
    }
  }
}
)
export class Webmessagetypes extends Entity {}

Then using lb4 repository i can select the created model with SPACE and then validate the creation of his corresponding repository file with ENTER.

But after creating the repository file, it has a typescript compile error Property 'id' does not exist on type 'Webmessagetypes'." in webmessagetypes.repository.ts:

export class WebmessagetypesRepository extends DefaultCrudRepository<
  Webmessagetypes,
  typeof Webmessagetypes.prototype.id // here
> {
  constructor(
    @inject('...') dataSource: DataSource,
  ) {
    super(Webmessagetypes, dataSource);
  }
}

I can share all the step i made through lb4 repository if necessary.

I think an automatic discovery command would be interesting as it would help migrating existing projects to lb4 and some people just prefer creating their database first.

PS: sorry for late answer

[DivCorleone]

My first workaround is working, so i'm closing the threat.

[bagintz]

I am having some trouble with lb4 discover, it is showing up views from the performance schema instead of tables from my DB. Where can I ask about this?

[dhmlau]

@bagintz, please open an issue on https://github.com/strongloop/loopback-next. I believe you're using mysql? please specify that info in the ticket too. thanks!