Opened 5 years ago

Closed 3 years ago

#7746 closed Feature/Enhancement Request (fixed)

Some debian packages I'd love to use on chelsea.mayfirst.org

Reported by: https://id.mayfirst.org/gripuqamweb Owned by: https://id.mayfirst.org/ross
Priority: Medium Component: Tech
Keywords: GRIP-UQAM python virtualenv chelsea.mayfirst.org Cc:
Sensitive: no

Description


Change History (30)

comment:1 Changed 5 years ago by https://id.mayfirst.org/gripuqamweb

Javascript

Using debian security updated files

  • tinymce (WYSIWYG editarea)
  • libjs-jquery-tablesorter

Generic

Django

Web application framework

  • python-django-south (schema and data migrations, for database safe deployment of new versions of websites)
  • python-django-social-auth (OpenID, etc.)
  • python-django-tinymce (WYSIWYG editarea)

SQLAlchemy

Database SQL toolkit and object-relational mapper (ORM)

  • alembic (schema and data migrations, for SQLAlchemy database safe deployment of new versions of websites)
  • python-elixir
  • python-sqlalchemy
  • python-sqlalchemy-doc (DB versioning /usr/share/doc/python-sqlalchemy-doc/examples/versioning)
  • python-zope.sqlalchemy (very recommended or needed by pyramid for sqlalchemy)

Pyramid

Minimalistic web framework

  • python-pyramid
  • python-pyramid-beaker (secure session factory)
  • python-pyramid-tm (recommended by python-pyramid)
  • python-weberror (recommended by python-pyramid)
  • python-webhelpers (recommended by python-pyramid)

Django - Pyramid comparison

Django Pyramid with SQLAlchemy
included python-sqlalchemy
included python-elixir
python-django-south alembic

Not available yet in debian

(So maybe it should go on another request), maybe I'll just use them without debian.

Broken or problematic debian packages

Other tickets for GRIP-UQAM

Last edited 5 years ago by https://id.mayfirst.org/gripuqamweb (previous) (diff)

comment:2 Changed 5 years ago by https://id.mayfirst.org/jamie

  • Owner set to https://id.mayfirst.org/ross
  • Status changed from new to assigned

Hey Ross - I'm assigning to you for review. I'm inclined to install all debian packages on on chelsea What do you think?

Packages not in Debian are probably best installed by the local user.

jamie

comment:3 Changed 5 years ago by https://id.mayfirst.org/gripuqamweb

I added http://packages.qa.debian.org/php-sabredav (not yet in debian stable).

comment:4 Changed 5 years ago by https://id.mayfirst.org/ross

Hi gripuqamweb,

I would like to get a little clarity on your need for all of these packages. It looks to me like some of these packages are used mostly for development purposes. Generally, most development work should be done in local contexts outside a shared environment.

I'm happy to get these packages installed for you, but would you mind confirming that your plan is to do development locally and deploy in the shared environment?

thanks,

~/ross

comment:5 Changed 5 years ago by https://id.mayfirst.org/gripuqamweb

Hi Ross,

Development is made on machines distant from Mayfirst and after deployed to mayfirst, but we may want to improve the web site and deploy successive versions.The main website is developed by a Django enthusiast and specialist with Django Tendenci, I hope it will include the Django Help Desk application, neither of the two are available in Debian. The main project is now hosted on a provisional server and waiting to be transfered to mayfirst. I also want to prioritize the availability of a separate Django Help Desk site to help the follow up of the main project. Django Help Desk highly recommend python-django-south. For some specialized functionalities not included in the main site, I will use python-pyramid with sqlalchemy because it looks to me more flexible and interesting in cases where a Django application does not already exist of for simple little sites.

I don't know if it would help to give more detail, package by package.

Have a nice day.

comment:6 Changed 5 years ago by https://id.mayfirst.org/ross

  • Resolution set to fixed
  • Status changed from assigned to closed

Okay, I've installed everything on your list that is packaged for debian, except python-formalchemy. Please let me know if you run into any problems.

~/ross

comment:7 Changed 5 years ago by https://id.mayfirst.org/gripuqamweb

Thank you very much!

comment:8 follow-up: Changed 5 years ago by https://id.mayfirst.org/gripuqamweb

  • Resolution fixed deleted
  • Status changed from closed to assigned
Last edited 5 years ago by https://id.mayfirst.org/gripuqamweb (previous) (diff)

comment:9 Changed 5 years ago by https://id.mayfirst.org/essais

  • Keywords GRIP-UQAM added

comment:10 in reply to: ↑ 8 Changed 5 years ago by https://id.mayfirst.org/essais

Replying to https://id.mayfirst.org/gripuqamweb:

Second wave

I just received a new list from the python devlopper of our main website. I am sorry to do this by etaps... (better wiki formatting)

comment:11 Changed 5 years ago by https://id.mayfirst.org/essais

Last edited 5 years ago by https://id.mayfirst.org/essais (previous) (diff)

comment:12 Changed 5 years ago by https://id.mayfirst.org/ross

  • Resolution set to fixed
  • Status changed from assigned to closed

Okay, these packages have been installed. Hopefully, that will get you what you need.

~/ross

comment:13 Changed 5 years ago by https://id.mayfirst.org/dkg

  • Resolution fixed deleted
  • Status changed from closed to assigned

Sorry, i just wanted to check in here: the python-* modules are all just normal software packages (utilities), but gunicorn actually also installs a service and tries to run it.

Is this service something we want or need? If so, how are configuration choices made for the running service? how do we avoid opening security holes by having a network-accessible service listening? If we don't want or need it as a system service, how are we explicitly ensuring that the service does not run automatically?

comment:14 Changed 5 years ago by https://id.mayfirst.org/ross

This is a good question. Looking at the configuration files for gunicorn, I don't see how the process could run without causing potential security holes and/or conflicts on standard listening ports for regular users.

Based on this observation, I'm uninstalling this package. If we can get some recommendations from the developer on how to run this securely in a shared environment, I'd be inclined to re-install it. Otherwise, it does seem the risks are too great.

comment:15 Changed 5 years ago by https://id.mayfirst.org/dkg

maybe we want to just disable the gunicorn service directly, using update-rc.d, or modifying /etc/init.d/gunicorn, or something similar, so that the toolset is available but the service is not?

I'm reminded of my old argument that tools should be distinct from services.

comment:16 Changed 5 years ago by https://id.mayfirst.org/ross

As it was, the service itself was not running. However, that fact did not seem to result in the tool being called. From what I could tell, based on some tests with a temporary hosting order, someone with the necessary configuration files could conceivably run gunicorn --config gun.conf myapp specifying 127.0.0.1:80 and interfere with port 80.

It is hopeful that such a port interference would not be allowed, but I lack the experience necessary to make that determination. Thus I'd prefer, until I have further evidence to leave the package uninstalled.

comment:17 Changed 5 years ago by https://id.mayfirst.org/jamie

I think it should be safe to leave the package, but configured not to start (I suggest: update-rc.d gunicorn disabled 2 to prevent it from starting in run level 2, our default run level). My understanding is:

  • All services specified to start in /etc/rc2.d/ should start before any regular user has a chance to start a service.
  • If a user tries to start a service on a port that is currently in use, it will fail
  • Only root users can start services that listen on ports less than 1024

comment:18 follow-up: Changed 5 years ago by https://id.mayfirst.org/essais

Thanks for all work you did, really appreciated. For gunicorn, python-django-tendenci depends on it and installs it in its virtualenv if not on the system, but my impression is that it is mostly used with nginx installations.

I cannot use python-virtualenv:

$ virtualenv --system-site-packages <dir>

makes an error. python-virtualenv recommends python-pip which is not installed. python-pip is also recommended by https://tendenci.readthedocs.org/en/latest/installation/remote.html

Can I ask you to install python-pip?

comment:19 in reply to: ↑ 18 Changed 5 years ago by https://id.mayfirst.org/dkg

Replying to https://id.mayfirst.org/essais:

Thanks for all work you did, really appreciated. For gunicorn, python-django-tendenci depends on it and installs it in its virtualenv if not on the system, but my impression is that it is mostly used with nginx installations.

If it is not used by the tool in some common modes of deployment, why is it considered a dependency? that seems like an odd choice by the tendenci maintainers.

It sounds to me like we should install gunicorn but disable it in runlevel 2 at least. Reading the update-rc.d manpage suggests that jamie's suggestion is slightly wrong. I think it should be:

update-rc.d gunicorn disable 2

comment:20 Changed 5 years ago by https://id.mayfirst.org/essais

If I read #7841 and https://www.debian-administration.org/users/dkg/weblog/68 I need socat to access postgresql databases with pgadmin3. Could you please install socat and python-pip #comment:18 ?

Last edited 5 years ago by https://id.mayfirst.org/essais (previous) (diff)

comment:21 Changed 5 years ago by https://id.mayfirst.org/dkg

  • Resolution set to fixed
  • Status changed from assigned to feedback

I've just made it so that socat will be on all the machines once the next puppet tag goes out, and installed socat on chelsea.

I've installed gunicorn and configured it as recommended in comment:11:

0 chelsea:~# service gunicorn stop
[ ok ] Stopping Gunicorn workers:.
0 chelsea:~# update-rc.d gunicorn disable 2 3 4 5
update-rc.d: using dependency based boot sequencing
insserv: warning: current start runlevel(s) (empty) of script `gunicorn' overrides LSB defaults (2 3 4 5).
insserv: warning: current stop runlevel(s) (0 1 2 3 4 5 6) of script `gunicorn' overrides LSB defaults (0 1 6).
0 chelsea:~# 

Against my better judgement, I've gone ahead and installed python-pip on chelsea. I think that running a production service using virtualenv and locally-installed modules via pip is a bad idea, because it puts you in the position of needing to maintain the modules, rather than relying on the operating system and the system administrators to maintain the modules. But it's on chelsea now if you want it.

comment:22 Changed 5 years ago by https://id.mayfirst.org/essais

  • Resolution fixed deleted
  • Status changed from feedback to assigned

Thanks! I know this problem of maitaining modules is a big issue.

comment:23 Changed 5 years ago by https://id.mayfirst.org/dkg

  • Resolution set to fixed
  • Status changed from assigned to feedback

You've moved this ticket from the feedback state back into the assigned state, but i don't know what other changes you're looking for. Could you clarify what you need?

comment:24 Changed 5 years ago by https://id.mayfirst.org/essais

  • Resolution fixed deleted
  • Status changed from feedback to assigned

dkg sorry I did not notice I was reopening the ticket, I would have loved not to change anything regarding resolution, neither reopen, neither close, maybe I shoud have written nothing, which would have been a bad idea too.

BTW I am trying to use virtualenv on a brand new UNIX account and I get an error:

gripuqam-chelsea@chelsea:~/tendenci_5.1.233$ virtualenv --system-site-packages tendenci_5.1.233_venv/
New python executable in tendenci_5.1.233_venv/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...
  Error [Errno 13] Permission denied while executing command /home/members/gripuq...env/bin/easy_install /usr/share/python-vi...p-1.1.debian1.tar.gz
...Installing pip...done.
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 3, in <module>
    virtualenv.main()
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 938, in main
    never_download=options.never_download)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 1054, in create_environment
    install_pip(py_executable, search_dirs=search_dirs, never_download=never_download)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 643, in install_pip
    filter_stdout=_filter_setup)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 976, in call_subprocess
    cwd=cwd, env=env)
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
    raise child_exception
OSError: [Errno 13] Permission denied

I have to check if it is an error or a warning...

gripuqam-chelsea@chelsea:~/tendenci_5.1.233$ ls tendenci_5.1.233_venv/bin
easy_install  easy_install-2.7  python

On a debian 7.0 system I installed I have:

$ virtualenv --system-site-packages venv/
New python executable in venv/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.

$ ls venv/bin/
activate  activate.csh  activate.fish  activate_this.py  easy_install  easy_install-2.7  pip  pip-2.7  python

From these data, I guess something went wrong and some files are missing in the virtualenv!

comment:25 Changed 5 years ago by https://id.mayfirst.org/dkg

I'm not sure what's going wrong for you here, sorry.

for what it's worth, chelsea is running debian 7.2 (wheezy), so it's not clear to me what distinction you're making between the two examples.

btw, thanks for including these transcripts, they're very useful to be able to tell what is going on. terminal_transcripts has some recommendations about how to make them even more useful (in particular, including the return code in the prompt, and always showing the final trailing prompt would be great).

comment:26 Changed 5 years ago by https://id.mayfirst.org/dkg

  • Keywords python virtualenv chelsea.mayfirst.org added

comment:27 Changed 5 years ago by https://id.mayfirst.org/essais

It looks maybe like a strange python bug, so including return codes:

0 gripuqam-chelsea@chelsea:~$ virtualenv --system-site-packages tendenci_5.1.233_venv
New python executable in tendenci_5.1.233_venv/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
0 gripuqam-chelsea@chelsea:~$ cd tendenci_5.1.233
0 gripuqam-chelsea@chelsea:~/tendenci_5.1.233$ virtualenv --system-site-packages tendenci_5.1.233_venv
New python executable in tendenci_5.1.233_venv/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...
  Error [Errno 13] Permission denied while executing command /home/members/gripuq...env/bin/easy_install /usr/share/python-vi...p-1.1.debian1.tar.gz
...Installing pip...done.
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 3, in <module>
    virtualenv.main()
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 938, in main
    never_download=options.never_download)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 1054, in create_environment
    install_pip(py_executable, search_dirs=search_dirs, never_download=never_download)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 643, in install_pip
    filter_stdout=_filter_setup)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 976, in call_subprocess
    cwd=cwd, env=env)
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
    raise child_exception
OSError: [Errno 13] Permission denied
1 gripuqam-chelsea@chelsea:~/tendenci_5.1.233$ ls tendenci_5.1.233_venv/bin/
easy_install      easy_install-2.7  python            
1 gripuqam-chelsea@chelsea:~/tendenci_5.1.233$

Looks like it is related to the directory name!

comment:28 Changed 5 years ago by https://id.mayfirst.org/essais

Looks like _ and . are not always welcome in containing directory name and virtualenv directory name! Here are some other working examples:

0 gripuqam-chelsea@chelsea:~/tendenci_51233$ virtualenv --system-site-packages tendenci51233venv
New python executable in tendenci51233venv/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
0 gripuqam-chelsea@chelsea:~/tendenci_51233$

0 gripuqam-chelsea@chelsea:~/tendenci51233$ virtualenv --system-site-packages tendenci51233venv
New python executable in tendenci51233venv/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
0 gripuqam-chelsea@chelsea:~/tendenci51233$

1 gripuqam-chelsea@chelsea:~/tendenci51233$ virtualenv --system-site-packages tendenci_51233venv
New python executable in tendenci_51233venv/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
0 gripuqam-chelsea@chelsea:~/tendenci51233$
  • It gives me the impression I better avoid _ and . in containing and virtualenv directory names.
  • I cannot reproduce this bug on another machine with the same os version.

comment:29 Changed 5 years ago by https://id.mayfirst.org/dkg

Cool, sounds like you've got it sorted out and things are working. weird about the directory naming, though. have you found any upstream bug reports or notes or documentation that explain that limitation? If not, have you considered opening an issue with upstream ?

comment:30 Changed 3 years ago by https://id.mayfirst.org/gripuqam

  • Resolution set to fixed
  • Status changed from assigned to closed

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.