Page MenuHomePhabricator
Create Task
Maniphest T351940

Enable local runs for article-descriptions model
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

I want to refactor the article-descriptions model server so that I can run the server locally without using docker containers.

Details

SubjectRepoBranchLines +/-
ml-services: fix rest gateway endpoint creation in article descriptionsoperations/deployment-chartsmaster+1 -1
article-descriptions: add helper function for rest gateway urlmachinelearning/liftwing/inference-servicesmain+77 -10
ml-services: update article-desc imageoperations/deployment-chartsmaster+1 -1
article-descriptions: fix boolean parsing of env varmachinelearning/liftwing/inference-servicesmain+2 -2
article-descriptions: enable local runmachinelearning/liftwing/inference-servicesmain+79 -29
ml-services: renamed directory for article_descriptionsintegration/configmaster+2 -2
article-descriptions: fix AsyncSession host headermachinelearning/liftwing/inference-servicesmain+3 -3
Customize query in gerrit

Related Objects

Event Timeline

isarantopoulos created this task.Nov 24 2023, 5:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 24 2023, 5:04 PM
gerritbot added a comment.Nov 24 2023, 5:05 PM

Change 976670 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[machinelearning/liftwing/inference-services@main] article-descriptions: enable local run

https://gerrit.wikimedia.org/r/976670

gerritbot added a project: Patch-For-Review.Nov 24 2023, 5:05 PM
isarantopoulos added a comment.Nov 27 2023, 8:45 AM

Current status is that I'm able to start the model server but I'm getting an error when making a request.

2023-11-24 21:04:56.674 uvicorn.error ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 419, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/timing_asgi/middleware.py", line 68, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/kserve/protocol/rest/v1_endpoints.py", line 76, in predict
    response, response_headers = await self.dataplane.infer(model_name=model_name,
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/kserve/protocol/dataplane.py", line 311, in infer
    response = await model(request, headers=headers)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/kserve/model.py", line 108, in __call__
    payload = await self.preprocess(body, headers) if inspect.iscoroutinefunction(self.preprocess) \
  File "/Users/isaranto/repoz/inference-services/article_descriptions/model_server/model.py", line 47, in preprocess
    descriptions, sitelinks, blp = await self.get_wikidata_info(lang, title)
  File "/Users/isaranto/repoz/inference-services/article_descriptions/model_server/model.py", line 193, in get_wikidata_info
    session = mwapi.AsyncSession(
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/mwapi/async_session.py", line 53, in __init__
    setattr(self.session, key, value)
AttributeError: can't set attribute
isarantopoulos updated the task description. (Show Details)Nov 27 2023, 9:03 AM
gerritbot added a comment.Nov 27 2023, 3:01 PM

Change 977241 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] article-descriptions: fix AsyncSession host header

https://gerrit.wikimedia.org/r/977241

kevinbazira subscribed.EditedNov 27 2023, 3:02 PM

I discovered what was causing this issue and pushed a patch for it here.

Essentially the first code snippet below is throwing this error AttributeError: can't set attribute but the second code snippet works well.

session = mwapi.AsyncSession(
    host=self.wiki_url,
    user_agent=self.user_agent,
    session=self.get_http_client_session("mwapi"),
    headers={"Host": "www.wikidata.org"},
)
session = mwapi.AsyncSession(
    host=self.wiki_url,
    user_agent=self.user_agent,
    session=self.get_http_client_session("mwapi"),
)
session.headers["Host"] = "www.wikidata.org"
gerritbot added a comment.Nov 27 2023, 4:17 PM

Change 977241 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] article-descriptions: fix AsyncSession host header

https://gerrit.wikimedia.org/r/977241

kevinbazira mentioned this in rMLISc46ed2a8e026: article-descriptions: fix AsyncSession host header.Nov 27 2023, 4:17 PM
isarantopoulos added a comment.Nov 28 2023, 7:47 AM

Currently investigating the following error I get on model load when setting low_cpu_usage=True :

 File "/Users/isaranto/repoz/inference-services/article_descriptions/model_server/model.py", line 301, in <module>
    model = ArticleDescriptionsModel(model_name)
  File "/Users/isaranto/repoz/inference-services/article_descriptions/model_server/model.py", line 30, in __init__
    self.load()
  File "/Users/isaranto/repoz/inference-services/article_descriptions/model_server/model.py", line 33, in load
    self.model.load_model(self.model_path)
  File "/Users/isaranto/repoz/inference-services/article_descriptions/model_server/utils.py", line 101, in load_model
    model = model.to(device)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2271, in to
    return super().to(*args, **kwargs)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1145, in to
    return self._apply(convert)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 797, in _apply
    module._apply(fn)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 797, in _apply
    module._apply(fn)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 820, in _apply
    param_applied = fn(param)
  File "/Users/isaranto/repoz/inference-services/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1143, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
NotImplementedError: Cannot copy out of meta tensor; no data!

Moreover model transfer to device (GPU) should take place after the web server's initialization as described and done in the [[ https://gerrit.wikimedia.org/r/plugins/gitiles/machinelearning/liftwing/inference-services/+/refs/heads/main/llm/model-server/model.py#56 | check_gpu function in llm models ]].

isarantopoulos set the point value for this task to 2.Nov 28 2023, 2:52 PM
kevinbazira mentioned this in T348156: Goal: Increase the number of models hosted on Lift Wing.Nov 28 2023, 3:40 PM
calbon triaged this task as Medium priority.Nov 28 2023, 3:51 PM
calbon moved this task from Unsorted to In Progress on the Machine-Learning-Team board.
isarantopoulos claimed this task.Nov 28 2023, 3:51 PM
gerritbot added a comment.Nov 29 2023, 1:24 PM

Change 978535 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[integration/config@master] ml-services: renamed directory for article_descriptions

https://gerrit.wikimedia.org/r/978535

gerritbot added a comment.Nov 29 2023, 5:36 PM

Change 978535 merged by jenkins-bot:

[integration/config@master] ml-services: renamed directory for article_descriptions

https://gerrit.wikimedia.org/r/978535

Stashbot added a comment.Nov 29 2023, 5:43 PM

Mentioned in SAL (#wikimedia-releng) [2023-11-29T17:43:05Z] <hashar> Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/978535 # T351940

gerritbot added a comment.Nov 30 2023, 2:18 PM

Change 976670 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] article-descriptions: enable local run

https://gerrit.wikimedia.org/r/976670

isarantopoulos mentioned this in rMLIS88d0063c6528: article-descriptions: enable local run.Nov 30 2023, 2:19 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 30 2023, 2:30 PM
gerritbot added a comment.Nov 30 2023, 2:37 PM

Change 979111 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[machinelearning/liftwing/inference-services@main] article-descriptions: fix boolean parsing of env var

https://gerrit.wikimedia.org/r/979111

gerritbot added a project: Patch-For-Review.Nov 30 2023, 2:37 PM
gerritbot added a comment.Nov 30 2023, 2:44 PM

Change 979114 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[operations/deployment-charts@master] ml-services: update article-desc image

https://gerrit.wikimedia.org/r/979114

gerritbot added a comment.Nov 30 2023, 4:58 PM

Change 979111 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] article-descriptions: fix boolean parsing of env var

https://gerrit.wikimedia.org/r/979111

isarantopoulos mentioned this in rMLIS65538428100b: article-descriptions: fix boolean parsing of env var.Nov 30 2023, 4:59 PM
gerritbot added a comment.Nov 30 2023, 5:13 PM

Change 979114 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: update article-desc image

https://gerrit.wikimedia.org/r/979114

isarantopoulos added a comment.EditedNov 30 2023, 5:26 PM

After deploying the changes in this task I'm getting a 500 with the following error logs

Traceback (most recent call last):
  File "/opt/lib/python/site-packages/uvicorn/protocols/http/httptools_impl.py", line 419, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/opt/lib/python/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/opt/lib/python/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/lib/python/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/lib/python/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/opt/lib/python/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/opt/lib/python/site-packages/timing_asgi/middleware.py", line 70, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/opt/lib/python/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/opt/lib/python/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/opt/lib/python/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/opt/lib/python/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/opt/lib/python/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/opt/lib/python/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/opt/lib/python/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/opt/lib/python/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "/opt/lib/python/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "/opt/lib/python/site-packages/kserve/protocol/rest/v1_endpoints.py", line 76, in predict
    response, response_headers = await self.dataplane.infer(model_name=model_name,
  File "/opt/lib/python/site-packages/kserve/protocol/dataplane.py", line 311, in infer
    response = await model(request, headers=headers)
  File "/opt/lib/python/site-packages/kserve/model.py", line 122, in __call__
    else self.predict(payload, headers)
  File "/srv/article_descriptions/model_server/model.py", line 92, in predict
    prediction = self.model.predict(
  File "/srv/article_descriptions/model_server/utils.py", line 142, in predict
    tokens = self.model.generate(
  File "/opt/lib/python/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/srv/article_descriptions/model_server/descartes/src/models/descartes_mbart.py", line 570, in generate
    batch_size = inputs_tensor[list(inputs_tensor.keys())[0]].shape[0]
AttributeError: 'NoneType' object has no attribute 'shape'
INFO:root:Opening a new Asyncio session for restgateway.
ERROR:root:Failed to retrieve first paragraph: 404, message='Not Found', url=URL('http://api-ro.discovery.wmnet/v1/page/summary/Clandonald')
INFO:root:Opening a new Asyncio session for restgateway.
ERROR:root:Failed to retrieve first paragraph: 404, message='Not Found', url=URL('http://api-ro.discovery.wmnet/v1/page/summary/Clandonald')

Reverted the deployment in order to investigate and fix.

Maintenance_bot removed a project: Patch-For-Review.Nov 30 2023, 5:30 PM
isarantopoulos added a comment.Dec 1 2023, 9:08 AM

The above issue was caused by the folowing line of code

base_url = urljoin(self.rest_gateway_endpoint, self.wiki_url or mw_host)

it tries to create a url from rest_gateway_endpoint and wiki_url, so http://rest-gateway.discovery.wmnet:411 and http://api-ro.discovery.wmnet respectively. The result of urljoin ends up being just the latter cause it expects a base url and a secondary url to perform a jon More info in the docs

However in this line of code we should be merging with mw_host instead and in this case we should prepend the scheme (https://) when we are not using the rest_gateway_endpoint.
I'll be adding a function and a couple of unit tests to the code to make sure it behaves properly for all scenarios.

gerritbot added a comment.Dec 1 2023, 3:02 PM

Change 979369 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[machinelearning/liftwing/inference-services@main] article-descriptions: add helper function for rest gateway url

https://gerrit.wikimedia.org/r/979369

gerritbot added a project: Patch-For-Review.Dec 1 2023, 3:02 PM
gerritbot added a comment.Dec 4 2023, 4:47 PM

Change 979369 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] article-descriptions: add helper function for rest gateway url

https://gerrit.wikimedia.org/r/979369

isarantopoulos mentioned this in rMLIS4ed330dea651: article-descriptions: add helper function for rest gateway url.Dec 4 2023, 4:48 PM
gerritbot added a comment.Dec 4 2023, 4:54 PM

Change 980002 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[operations/deployment-charts@master] ml-services: fix rest gateway endpoint creation in article descriptions

https://gerrit.wikimedia.org/r/980002

gerritbot added a comment.Dec 5 2023, 8:58 AM

Change 980002 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: fix rest gateway endpoint creation in article descriptions

https://gerrit.wikimedia.org/r/980002

Maintenance_bot removed a project: Patch-For-Review.Dec 5 2023, 9:10 AM
isarantopoulos added a comment.Dec 5 2023, 1:10 PM

The model can now be ran locally following the instructions in the README.md file.
I added a couple of unit tests that assess that the correct url is created for the REST API requests.
I ran into some issues with virtualenv and tox while trying to add the unit tests to run in the test image so I left it for now.
I suggest we revisit how our whole test images run when we move the repo to Gitlab.

isarantopoulos closed this task as Resolved.Dec 5 2023, 2:35 PM
isarantopoulos set Final Story Points to 5.
isarantopoulos moved this task from In Progress to 2023-2024 Q3 Done on the Machine-Learning-Team board.Dec 5 2023, 2:38 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL