API:
REST API Endpoints:
1. GET/POST /v1/{project}/prefix/{prefix}/classes
List of classes
2. GET/POST /v1/{project}/prefix/{prefix}/parameters
Hiera specification for this node
3. GET /v1/{project}/node/{fqdn}
List of classes
Hiera specification for this node
4. GET /v1/{project}/prefix
List of prefix mappings| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew | T42525 Cant add a security group to an existing instance | |||
| Resolved | Andrew | T87279 Make OpenStack Horizon useful for production labs | |||
| Resolved | Andrew | T91990 Horizon dashboard for managing instance puppet config | |||
| Resolved | yuvipanda | T133412 Create puppet backend with REST api for labs instance configuration |
Change 285014 had a related patch set uploaded (by Yuvipanda):
[WIP] MySQL backend for storing roles / hiera data for labs
Change 285014 merged by Andrew Bogott:
MySQL backend for storing roles / hiera data for labs
Change 294510 had a related patch set uploaded (by Andrew Bogott):
Move labspuppetbackend to port 8100; open firewall
Change 294510 merged by Andrew Bogott:
Move labspuppetbackend to port 8100; open firewall
The backend frequently errors out, like this:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/flask/app.py", line 1836, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1820, in wsgi_app response = self.make_response(self.handle_exception(e)) File "/usr/lib/python3/dist-packages/flask/app.py", line 1403, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/flask/_compat.py", line 33, in reraise raise value File "/usr/lib/python3/dist-packages/flask/app.py", line 1817, in wsgi_app response = self.full_dispatch_request() File "/usr/lib/python3/dist-packages/flask/app.py", line 1477, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/lib/python3/dist-packages/flask/app.py", line 1381, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/flask/_compat.py", line 33, in reraise raise value File "/usr/lib/python3/dist-packages/flask/app.py", line 1473, in full_dispatch_request rv = self.preprocess_request() File "/usr/lib/python3/dist-packages/flask/app.py", line 1666, in preprocess_request rv = func() File "/usr/local/lib/python3.4/dist-packages/labspuppetbackend.py", line 18, in before_request passwd=os.environ['MYSQL_PASSWORD'] File "/usr/lib/python3/dist-packages/pymysql/__init__.py", line 88, in Connect return Connection(*args, **kwargs) File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 679, in __init__ self.connect() File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 891, in connect self._request_authentication() File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1097, in _request_authentication auth_packet = self._read_packet() File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 966, in _read_packet packet.check_error() File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 394, in check_error err.raise_mysql_exception(self._data) File "/usr/lib/python3/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception _check_mysql_exception(errinfo) File "/usr/lib/python3/dist-packages/pymysql/err.py", line 112, in _check_mysql_exception raise errorclass(errno, errorvalue)
pymysql.err.OperationalError: (1040, 'Too many connections')
[pid: 18405|app: 0|req: 29/29] 208.80.153.14 () {28 vars in 553 bytes} [Fri Jun 17 19:50:04 2016] GET /v1/labtestproject/prefix/puppettest104.labtestproject.codfw.testlabs/classes => generated 0 bytes in 17 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)
Change 296830 had a related patch set uploaded (by Yuvipanda):
labspuppetbackend: Use sqlalchemy for connection pooling
The gui prototype is on labtesthorizon.wikimedia.org and the code is on labtestweb2001.wikimedia.org in /usr/lib/python2.7/dist-packages/wikimediapuppettab. It'll take me a bit to remember what I was doing to produce those errors :)
Scratch that -- that 'too many connections' error with mysql is a thing I've been seeing with other services as well. So I'm pretty sure it's not a problem with Yuvi's code. Disregard for now.
This server gets/sends hiera json, but presumably the user wants to edit yaml. I can convert back and forth in the client but formatting and comments will be lost, which is not great.
After discussion with yuvi, we're going to try making the API yaml-only. Yuvi will update the API when he has a minute.
The API as described in the description does not at all correspond to the actual code. I'm pretty sure that we say 'roles' and 'hiera' now instead of 'classes' and 'parameters.'
Using a single pathway for hiera means that I have to comingle two different sources of hiera yaml:
Since pyYaml scrambles formatting on the round-trip, this is pretty bad news for the usability of 1). I'm not sure if we should just accept the scrambling or split out the API so it supports both pathways as separate urls.
So far I'm finding the format-scrambling issue to be tolerable, so best not to waste much time on this until I have a more complete demo running.
@yuvipanda, the puppet gui running on labtesthorizon should now be fully functional. It needs more code review but what you see there should work well enough for you to exercise your API and get it actually hooked up to the puppetmaster.
Change 310619 had a related patch set uploaded (by Yuvipanda):
labs: Introduce httpyaml backend for labtest
Change 310669 had a related patch set uploaded (by Yuvipanda):
labtest: actually add httpyaml backend
Right now I can add new prefixes (by setting a role list to []) but I need a new API call to delete a prefix.
Change 310879 had a related patch set uploaded (by Andrew Bogott):
labspuppetbackend: Add prefix delete
Change 310958 had a related patch set uploaded (by Andrew Bogott):
labspuppetbackend: fixed mysql typo
Change 311742 had a related patch set uploaded (by Andrew Bogott):
Set CHARSET=utf8mb4 for the labspuppetbackend tables.
Change 311742 merged by Andrew Bogott:
Set CHARSET=utf8mb4 for the labspuppetbackend tables.
Change 296830 abandoned by Yuvipanda:
labspuppetbackend: Use sqlalchemy for connection pooling