Okuna

The API server for Okuna.

Table of contents

• Requirements
• Project overview
• Contributing
  • Code of Conduct
  • License
  • Responsible disclosure
  • Other issues
  • Git commit message conventions

• Getting started
  • Clone the repository
  • Meet the Okuna CLI
  • Okuna CLI Operational Modes
  • Full mode
    • Installation
    • Available commands
      • up-full
      • down-full

  • Services-only mode
    • Installation
    • Available commands
      • up-services-only
      • down-services-only

    • Running the Okuna API server locally
  • Available test data
  • Other Okuna CLI commands
    • clean

  • Okuna CLI behind the scenes
    • docker-compose
    • Environment files

• Available Django commands
  • Official Django commands
    • manage.py migrate
    • manage.py collectmedia
    • manage.py loaddata

  • Custom django commands
    • manage.py create_invite
    • manage.py import_invites
    • manage.py reset_invite_email_boolean
    • manage.py send_invites
    • manage.py create_post_media_thumbnails
    • manage.py migrate_post_images
    • manage.py import_proxy_blacklisted_domains
      • Example

    • manage.py flush_proxy_blacklisted_domains
    • manage.py worker_health_check
    • Crowdin translations update
• Available Django jobs
  • openbook_posts.jobs.flush_draft_posts
  • openbook_posts.jobs.curate_top_posts
  • openbook_posts.jobs.clean_top_posts

• Translations
• FAQ
  • Double logging in console

• Pipenv
• MySQL
• Redis
• libmagic
• ffmpeg

Project overview

The project is a Django application.

Contributing

There are many different ways to contribute to the website development, just find the one that best fits with your skills and open an issue/pull request in the repository.

Examples of contributions we love include:

• Code patches
• Bug reports
• Patch reviews
• Translations
• UI enhancements

Code of Conduct

Please read and follow our Code of Conduct.

License

Every contribution accepted is licensed under MIT or any later version. You must be careful to not include any code that can not be licensed under this license.

Please read carefully our license and ask us if you have any questions.

Responsible disclosure

Cyber-hero? Check out our Vulnerability Disclosure page.

Other issues

We're available almost 24/7 in the Okuna slack channel. Join us!

Git commit message conventions

Help us keep the repository history consistent 🙏!

We use gitmoji as our git message convention.

If you're using git in your command line, you can download the handy tool gitmoji-cli.

Getting started Clone the repository

Run the following in your command line

git clone git@github.com:OkunaOrg/okuna-api.git && cd okuna-api

The Okuna CLI is built to run a development instance of Okuna loaded with some test data, with a single command.

Okuna CLI Operational Modes

You can use the CLI in two modes.

1. Full mode - Best for Okuna mobile/web app development
2. Services-only mode - Best for Okuna API development

Depending on the kind of development you would like to do, follow the instructions below for your chosen mode.

Full mode

Best for Mobile/web app development

This mode brings a whole Okuna instance up, ready to use with a local Okuna mobile/web app.

Installation

Make sure the following are installed

• Python 3.5+
• docker-compose

Install the Okuna CLI python packages

pip install -r requirements-cli-only.txt

Starts Okuna

Run the following in your terminal

python okuna-cli.py up-full

🥳 Congrats, you should now have both the whole of Okuna running on port 80.

down-full

Shuts Okuna down

When existing the command that starts Okuna by pressing CTRL + C / CMD + C, Okuna will also be stopped.

If the process was abruptly terminated and Okuna is still running in the background you can also run

python okuna-cli.py down-full

Best for API development

The Okuna services are a SQL server, a Redis server, a job scheduler server and a job worker server.

This mode brings these services up but not the Okuna API itself, you are to run the API locally instead for a better development experience.

Installation

Make sure the following are installed

• Python 3.5+
• docker-compose

Install the Okuna python packages

pip install -r requirements.txt

Make the bootstrap_development_data.sh executable

chmod ./utils/scripts/bootstrap_development_data.sh +x

Starts the Okuna services

Run the following in your terminal

python okuna-cli.py up-services-only

Stops the Okuna services

When existing the command that starts Okuna by pressing CTRL + C / CMD + C, Okuna will also be stopped.

If the process was abruptly terminated and Okuna is still running in the background you can also run

python okuna-cli.py down-services-only

Once the Okuna services are up (using the up-services-only command), start the Okuna API locally by running

python manage.py runserver

This will start the API server on localhost:8000.

If you would like to expose the server on your network, for example, for testing the API server on your mobile device connected to the same network, run the following instead.

python manage.py runserver 0.0.0.0:8000

🥳 Congrats, you should now have both the Okuna services and the Okuna API running in sync.

Available test data

Within the provisioned Okuna development instance, you will find some test accounts and respective posts/communities.

The credentials for these accounts are the following.

username:martha
password:changeme123!

username:mike
password:changeme123!

username:miguel
password:changeme123!

** Cleans everything related to the Okuna CLI**

This will dispose of the employed database and generated config files.

Use this if you want to get a fresh version of Okuna next time you use the Okuna CLI up commands.

python okuna-cli.py clean

This section will try to demistify what the Okuna CLI does.

docker-compose

Behind the scenes, the Okuna CLI uses docker-compose to spawn and coordinate the following docker services.

• webserver - A server running the Okuna API
• db - A server with a MariaDB database
• redis - A server with a Redis database
• scheduler - A server responsible for running the scheduled Okuna jobs such as curating Explore/Top posts.
• worker - A server responsible for processing the Okuna jobs such as publishing a post or curating posts.

On services-only mode, the webserver is not spawned.

Environment files

When starting okuna-cli for the first time, 3 files will be generated

• .okuna-cli.json - Contains a flag indicating whether the Okuna instance was bootstrapped and the secret keys employed to kickstart all other services
• .env - The environment file used when running the Okuna CLI in services-only mode.
• .docker-compose.env - The environment file used in all Okuna docker compose services

Available Django commands

A list of official and custom django commands that might come handy.

If running the API locally you can execute them as

python manage.py $command

If running the API with the Okuna CLI full mode, you can execute them by connecting to the webserver machine by running

docker-compose -f docker-compose-full.yml exec webserver "/bin/bash"

Inside the machine you can then execute the commands as

python manage.py $command

Run the database migrations

python manage.py migrate

Collect the media fixtures

python manage.py collectmedia

Load the given data fixture file into the database

python manage.py loaddata circles.json emoji-groups.json emojis.json badges.json categories.json languages.json

Creates a user invite and outputs its token. Required for creating a new account.

usage: manage.py create_invite [-h] [--email EMAIL] [--username USERNAME] [--name NAME] [--badge BADGE]

Imports user invites from a kickstarter/indiegogo csv

usage: manage.py import_invites [-h] [--indiegogo PATH_TO_CSV] [--kickstarter PATH_TO_CSV]

Resets invite_email_sent boolean for un-used invites created in the last --days

usage: manage.py reset_invite_email_boolean [-h] [--days DAYS]

Send invite emails to all user invites who have not been sent the email.

usage: manage.py send_invites [-h]

####manage.py allocate_invites

Assign user invites to all or specific users.

usage: manage.py allocate_invites [-h] [--count INCREMENT_INVITES_BY_COUNT --limit [INVITE_COUNT_UPPER_LIMIT]] [--total TOTAL_INVITE_COUNT_TO_SET] [--username USERNAME]

*--limit works only with --count

manage.py create_post_media_thumbnails

Creates media_thumbnail, media_height and media_width for items which don't have it using the Post -> PostMedia relationship.

The command was created as a one off migration tool.

usage: manage.py create_post_media_thumbnails

Migrates Post -> PostImage to Post -> PostMedia -> PostImage.

The command was created as a one off migration tool.

manage.py import_proxy_blacklisted_domains

Import a list of domains to be blacklisted when calling the ProxyAuth and ProxyDomainCheck APIs.

usage: manage.py import_proxy_blacklisted_domains [--file The path to the file with the domains to import]

python manage.py import_proxy_blacklisted_domains --file ./openbook_common/misc/domain_blacklists/porn/domains

Flush all of the blacklisted proxy domains

usage: manage.py flush_proxy_blacklisted_domains

A a Django management command available for checking the worker health:

Each queue has a required configurable treshold. These are configured in the Django settings. The FAILED_JOB_THRESHOLD is the maximum amount of failed jobs that are allowed, before an alert is sent using the openbook_common.helpers.send_alert_to_channel command, which sends an alert to a monitoring channel on i.e. Slack. Using the ALERT_HOOK_URL option in the Django settings file, it is possible to add the Slack hook URL.

Other thresholds that are included are ACTIVE_JOB_THRESHOLD and ACTIVE_WORKER_THRESHOLD. Just as with the limit on failed jobs, there is an active job and worker limit too.

In openbook_common.utils.rq_helpers there is also a FailedRQJobs class, which has a function for removing failed jobs from the queue.

It is recommended to schedule the worker monitoring functions, to run at a 5 minute interval using crontab. Please DO NOT run the job as the root user.

Crowdin translations update

Download the latest django.po files in the respective locale/ folders from crowdin. Then locally run all or some of these commands depending on which models need updating. It will update the .json files, then check them in.

./manage.py shell < openbook_common/i18n/update_translations_emoji_groups.py
./manage.py shell < openbook_common/i18n/update_translations_emojis.py
./manage.py shell < openbook_moderation/i18n/update_translations.py
./manage.py shell < openbook_categories/i18n/update_translations.py

To schedule a job, go to the /admin/scheduler route on the running webserver.

The available jobs are

openbook_posts.jobs.flush_draft_posts

Cleans up all draft posts which have not being modified for a day.

Should be run every hour or so.

openbook_posts.jobs.curate_top_posts

Curates the top posts, which end up in the explore tab.

Should be run every 5 minutes or so.

openbook_posts.jobs.clean_top_posts

Cleans the top posts which should no longer be top posts.

This happens if an item is soft deleted, reported and approved

Should be run every 5 minutes or so.

Translations

1. Use ./manage.py makemessages -l es to generate messages. Doesn't matter which language we target, the translation tool is agnostic.
2. This generates a new django.po file in the locale/es/LC_MESSAGES/ folder. It will have all the source strings and a place to enter the translation strings. It doesnt overwrite previous translations.
3. Sometimes, if django is confused, it marks a string as fuzzy. So do search for the word 'fuzzy' in django.po
4. If you find a fuzzy string, you can resolve it manually. Finally each string should like this

#: openbook_lists/validators.py:10   <- place where the string occurs in code
msgid "The list does not exist."     <- english translations

msgstr "Die Liste ist nicht vorhanden."   <-- this will be empty for new strings

1. Upload this django.po file to https://crowdin.com/project/okuna/settings#files by pressing Update next to the existing django.po file.
2. Once all language volunteers have translated the new strings, download all the django.po files for each locale and put them in their respective folders.
3. Run ./manage.py compilemessages to auto-generate django.mo files.
4. You need to checkin both django.po and django.mo files for each locale.

FAQ Double logging in console

The local development server runs a separate process for the auto-reloader. You can turn off the auto-reload process by passing the --noreload flag.

python manage.py runserver --noreload