[WMDE-Fundraising] Epic: Rewrite of frontend
Open, NormalPublic

Description

Tracking issue for the WMDE Fundraising frontend "rewrite".

This "rewrite" will happen by starting from scratch architecture and design wise. As top down implementation is done, the corresponding parts of the old codebase can be used as they are, used in a refactored form, or thrown away and replaced entirely as we see fit.

Proposed approach

First we identify a use case that we predict is easy to implement and minimally depends on behaviour we'd need to implement for another one. We can then implement this use case in top down manner, creating the needed services, entities and other stuff as we see fit. This is where we decide for each part of the implementation if we simply use some old code, use a refactored version of some old code, or start from scratch. Once the use case is completed, we repeat the process, until we have all use cases (and thus completed both the red and yellow circles in this diagram). The next step then is to decide how we want to proceed with the presentation mechanism(s).

(https://www.youtube.com/watch?v=Nsjsiz2A9mg, https://blog.8thlight.com/assets/posts/2012-08-13-the-clean-architecture/CleanArchitecture-81565aba46f035911a5018e77a0f2d4e.jpg)

Use case identification

Visualization of conceptual components and their dependencies in the current codebase:

Identified use cases:

Display specific page from the CMS

Which page is specified in url, CMS may return content not found
GET request
URL: https://test.wikimedia.de/spenden/PAGENAME
new URL will probably something like https://test.wikimedia.de/page/PAGENAME

Validate Email

Validate email structure & do MX check
GET request
URL: https://test.wikimedia.de/ajax.php?module=action&action=checkEmail&eaddr=gb@blah.com
Success response: {"status":"OK"}
Error response: {"status":"ERR"}

Validate Bank Account

Validate bank data and return full set of bank data (converted from input)
GET request
URL for IBAN: https://test.wikimedia.de/ajax.php?module=action&action=checkIban&iban=DE12500105170648489890
URL for acct. number: https://test.wikimedia.de/ajax.php?module=action&action=generateIban&bankCode=50010517&accNum=0648489890
Success response {"bic":"INGDDEFFXXX","iban":"DE12500105170648489890","account":"0648489890","bankCode":"50010517","bankName":"ING-DiBa","status":"OK"}
Error response: {"status":"ERR"}

Add Donation

Add entry to donation table and send confirmation email. Redirect to appropriate payment processor if needed.
Verify the request's integrity and check for irregularities (i. e. bad words, exceeding donation amount limit). Refuse insertion of data if integrity check fails, set status to pending on irregularities. Must handle locale/country-dependent data (eg. different number format with comma as decimal separator, different ZIP code lengths depending on country, etc).
POST request, application/x-www-form-urlencoded
URL https://test.wikimedia.de/spenden/spende.php
Request Body: betrag_auswahl=5%2C00&betrag=&intervalType=0&zahlweise=BEZ&debit-type=non-sepa&iban=DE12500105170648489890
&bic=INGDDEFFXXX&konto=0648489890&blz=50010517&adresstyp=person&anrede=Herr&titel=Kein+Titel&vorname=jhebk&nachname=veebwk&firma=&strasse=mevbek3b2hn&plz=11234&ort=Berlin&country=DE&email=gb%40blah.com
&go_prepare--pay%3Aeinzug=&periode=0&bankname=ING-DiBa&iframe=&form=10h16_Confirm&appeal=&from_form=10h16_Form2_Ticker_Intrvar
&tracking=&source=web&referrer=https%3A%2F%2Ftest.wikimedia.de%2Fspenden%2F&wikilogin=&layout=form%3D10h16_Form1
&color=&impCount=0&bImpCount=0&skin=10h16&from_pay=&utoken=1ad43457353b285d23b109df379424b80f000d46&stamp=2016-01-04+15%3A52%3A45&PHPSESSID=nsh4hui3055nl76v9873invn33&appealExpanded=&back_form=10h16_Form1

Request model parameters

  • betrag
  • zahlweise
  • periode
  • iban
  • bic
  • konto
  • blz
  • bankname
  • adresstyp
  • anrede
  • titel
  • firma
  • vorname
  • nachname
  • firma
  • strasse
  • plz
  • ort
  • country
  • email

The required field set depends on the address type:

  • anonymous: all address fields can be left out
  • firma: vorname and nachname can be left out
  • person: firma can be left out

Additional parameters for the endpoint (presentation-related):

  • tracking (set of piwik campaign and keyword (e. g. campaignName/bannerName)
  • source (host name of referring site)
  • impCount (total impressions before donating)
  • bImpCount (current banner's impressions before donating)
  • form
  • from_form: current form
  • layout
  • color
  • skin
  • back_form: Form before the current form

Add Comment

Append a comment to the donation data.

  • All payment types except UEB (bank transfer), because we want to have a validated payment.
  • After a donation (but not membership), the user is being asked for a comment.
  • Can only be submitted

URL: https://test.wikimedia.de/ajax.php?module=action&action=addComment
Request body: kommentar=fjhwbjev+ejbjvbffjfhjf2hf2&eintrag=2&public=1&tracking=&source=web&referrer=https%3A%2F%2Ftest
.wikimedia.de%2Fspenden%2F&wikilogin=&layout=form%3D10h16_Form1&color=&impCount=0&bImpCount=0&skin=10h16
&token=1276888%2459b42194b31d0265df452735f6438a234bae2af8&sid=2343&utoken=b5b249c8beefb986faf8d186a3f16e86ef509ab3
&PHPSESSID=nsh4hui3055nl76v9873invn33
Error Response: {"status":"ERR","message":"Der Datensatz kann nicht mehr ver\u00e4ndert werden. Bitte wenden Sie sich an unsere Hotline."}
Success response: {"status":"OK","message":"Something"}
Relevant fields: kommentar, public, eintrag, token, utoken, sid

List public donations and comments

List comments in three formats: HTML page, RSS feed, JSON
Make name of anonymous donor translatable
GET request
Current URLs:

URLs apparently not used anymore:

json.php, html.php, and latest.php accept one request parameter:

  • n (number of entries to return)

json.php accepts additional parameters:

  • f (callback method wrapper for jsonp requests)
  • anon (include anonymous entries)

json.php Response:

  • betrag
  • spender
  • kommentar
  • datum
  • id

Parameters:

  • Paging (number of entries per page, page number or start date)
  • show anonymous comments yes/no

Add Membership Request

Generate entry in "request" table and send confirmation email
Verify the request's integrity and check for irregularities (i. e. bad words, exceeding membership fee limit). Refuse insertion of data if integrity check fails, set status to pending on irregularities.
URL: https://test.wikimedia.de/spenden/contact.php
POST request
Request Body:
membership_type=sustaining&adresstyp=person&anrede=Herr&titel=Kein+Titel&vorname=blah&nachname=blub&firma=&strasse=cnkevhwnw&plz=11207&ort=Berlin&country=DE&email=gb%40blah.com&dob=30.02.9999&phone=1234555&membership_fee_interval=12&membership_fee=25%2C00&membership_fee_custom=&debit-type=sepa&iban=DE12500105170648489890
&bic=INGDDEFFXXX&account_number=0648489890&bank_code=50010517&go_contact=&purpose=membership-full&back_form=Mitgliedschaft&member_agree=1&account_holder=blank&bank_name=ING-DiBa&wikilogin=
Request model parameters: All from request body except back_form and wikilogin

Add Subscription Request

Save personal data from a contact form in the "request" table (marked unconfirmed) and send a confirmation email with a link containing a unique ID.
Feature is partly implemented on wikipedia de and fundraising page, but not finished.
POST request
URL: https://test.wikimedia.de/ajax.php?module=action&action=subscribe

Confirm Subscription Request

Look up a user from the "request" table by unique ID and mark the entry as confirmed
GET request
URL: https://test.wikimedia.de/spenden/?action=confirmSubscription&id=UNIQUE_ID

Cancel Donation

mark donation as canceled and send email confirming the cancellation.
Can only be called if utoken is not expired.
URL: https://test.wikimedia.de/spenden/storno.php
Request model parameters:

  • token
  • sid
  • utoken

Handle Callback (PayPal Handler)

Handle all payment notification requests from PayPal.
POST request (must be validated, HTTP 406 error if other method was used)
URL: https://test.wikimedia.de/spenden/paypal_handler.php
Request model parameters:

Values that are being stored for documentation purpose only:

  • paypal_payer_id => payer_id
  • paypal_subscr_id => subscr_id
  • paypal_payer_status => payer_status
  • paypal_address_status => address_status
  • paypal_mc_gross => mc_gross
  • paypal_mc_currency => mc_currency
  • paypal_mc_fee => mc_fee
  • paypal_settle_amount => settle_amount

Values that are being stored for later export:

  • txn_id => ext_payment_id
  • payment_type => ext_payment_type
  • payment_status => ext_payment_status
  • payer_id => ext_payment_account
  • ext_payment_timestamp (generated at runtime)
  • subscr_id => ext_subscr_id

As the first step, the handler must post back the raw POST data (with the additional field cmd=_notify-validate) to the PayPal URL to signal that the request was received (See https://developer.paypal.com/webapps/developer/docs/classic/ipn/integration-guide/IPNImplementation/#specs). If posting back was not successful (got no 200 OK back or response content is not VERIFIED), the script has to quit.

Now the script does different things, depending on payment_status (see below).

Pending

If we're using the PayPal sandbox, set to Processed, otherwise ignore. Check if the sandbox still does not send Processed, otherwise just ignore.

Completed and Processed

Check if a related donation exists. If not, create a new record from the PayPal data and process it.

If the donation id is not empty and handle according to status. Do nothing except logging for states that don't make sense. Create a new donation record (with data from the old record) when status is "B" (bezahlt) and txn_type is subscr_payment. Process (new) donation.

The payment processor does not check the utoken, only the token, because payments can come in up to 5 days after creating the donation.

Other status strings

Log and do nothing. In the future we should check with the fundraising team, which states make sense in our context and which are already handled by defaults and Backend Application actions.

See https://github.com/wmde/fundraising/issues/970#issuecomment-164474583

Handle Callback (Credit Card Handler)

Called by the payment provider to confirm a payment. Validates the submitted data updates the status of the donation and triggers the confirmation mail. Callbacks concerning failed transactions are currectly being ignored, but a log entry should probably be added to the data set.

The submitted data is validated against the data set using the primary key, utoken, and donation amount. The status is updated only if the three values match, otherwise the handler responds with an error message. An additional requirement is to determine the credit card's expiration date by using the payment provider's API.

GET request
URL: https://test.wikimedia.de/spenden/mcp-handler.php

Request model parameters:

  • function (billing for successful transactions; error for failed transactions)
  • amount (in cents)
  • auth (16B; hex representation)
  • customerId (20B; hex representation; only set on function=billing)
  • sessionId (20B; hex representation; only set on function=billing)
  • transactionId (e. g. spenden.wikimedia.de-IDv5mgzw10kp)
  • utoken (20B; hex representation; created when adding the donation data set and passed to the payment provider)
  • donation_id (primary key of the donation data set)
  • errorcode (as defined in techdoc.micropayment.de; only on function=error)
  • errormessage (as defined in techdoc.micropayment.de; only on function=error)
  • title (transaction description as printed on the invoice)
  • country (ISO 3166-1 alpha-2)
  • currency (ISO 4217)
  • token
  • testmode

The serialized data array in the data set is being updated with the passed values. The values are mapped to different keys, though:

Values that are being stored for documentation purpose only:

  • currency => mcp_currency
  • amount => mcp_amount
  • country => mcp_country
  • auth => mcp_auth
  • title => mcp_title
  • sessionId => mcp_sessionid

Values that are being stored for later export:

  • transactionId => ext_payment_id
  • function => ext_payment_type
  • status => ext_payment_status
  • customerId => ext_payment_account
  • mcp_cc_expiry_date (needs to be determined by requesting the payment provider's API)
  • ext_payment_timestamp (generated at runtime)

Remaining use cases to discuss

  • Confirm donation (direct debit only at the moment, should probably be handled by "Add Donation" use case instead)
  • List Recent Donations (for "Donation Liveticker")
    • list includes N recent donations
    • each entry consists of donation date and time, and donation amount
    • this list covers all donations, not only those that have a donor added a comment

The new code

So far the idea is to put the rewrite in a new git repository at https://github.com/wmde/FundraisingFrontend. This way we get rid of any issues present in the old git repository history and can easily see what has been "rewritten" and what still needs to be done. (The code already there is for exploration and demonstration purposes and is not intended to be used or seen as a replacement of the original code.)

Next steps

(In no particular order)

  • Implement "list comments" use case to get a better idea of the amount of work involved and the types of issues we'll run into (to be able to better estimate total project cost).
  • Describe use cases in more detail, figure out which ones actually describe multiple use cases and which ones we missed.

Related Objects

StatusAssignedTask
Resolvedgabriel-wmde
OpenNone
DeclinedNone
Resolvedgabriel-wmde
ResolvedTobi_WMDE_SW
ResolvedNone
Resolvedgabriel-wmde
OpenNone
Resolvedgabriel-wmde
Resolvedgabriel-wmde
DeclinedNone
DuplicateNone
Declinedgabriel-wmde
DeclinedNone
DeclinedNone
DeclinedNone
ResolvedTobi_WMDE_SW
ResolvedTobi_WMDE_SW
ResolvedNone
ResolvedTobi_WMDE_SW
ResolvedTobi_WMDE_SW
ResolvedNone
Resolvedgabriel-wmde
InvalidNone
ResolvedTobi_WMDE_SW
ResolvedTobi_WMDE_SW
ResolvedTobi_WMDE_SW
DuplicateNone
ResolvedJeroenDeDauw
ResolvedTobi_WMDE_SW
ResolvedTobi_WMDE_SW
ResolvedTobi_WMDE_SW
Resolvedgabriel-wmde
ResolvedTobi_WMDE_SW
ResolvedNone
ResolvedJeroenDeDauw
ResolvedTobi_WMDE_SW
InvalidNone
ResolvedTobi_WMDE_SW
ResolvedNone
ResolvedNone
ResolvedNone
ResolvedTobi_WMDE_SW
ResolvedTobi_WMDE_SW
ResolvedJeroenDeDauw
ResolvedTobi_WMDE_SW
ResolvedTobi_WMDE_SW
Resolvedkai.nissen
Resolvedgabriel-wmde
Resolvedgabriel-wmde
Resolvedgabriel-wmde
Resolvedgabriel-wmde
InvalidNone
ResolvedTobi_WMDE_SW
Resolvedkai.nissen
ResolvedTobi_WMDE_SW
Resolvedkai.nissen
ResolvedTobi_WMDE_SW
ResolvedTobi_WMDE_SW
ResolvedTobi_WMDE_SW
ResolvedJeroenDeDauw
JeroenDeDauw updated the task description. (Show Details)
JeroenDeDauw raised the priority of this task from to Needs Triage.
JeroenDeDauw added a project: TCB-Team.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 21 2015, 4:43 PM
This comment was removed by JeroenDeDauw.
JeroenDeDauw updated the task description. (Show Details)Sep 21 2015, 4:48 PM
JeroenDeDauw set Security to None.
JeroenDeDauw updated the task description. (Show Details)Sep 21 2015, 4:54 PM
JeroenDeDauw updated the task description. (Show Details)Sep 21 2015, 4:57 PM
JeroenDeDauw updated the task description. (Show Details)
JeroenDeDauw updated the task description. (Show Details)Sep 21 2015, 5:03 PM

A more up-to-date diagram:

WMDE-leszek updated the task description. (Show Details)Oct 2 2015, 10:20 AM
WMDE-leszek added a comment.EditedOct 2 2015, 10:23 AM

Having looked at various comments and donation data lists, I have figured out that there are actually two comment lists being used. As they differ in scope, and the data being listed, I reckon they are separate use cases (please feel free to change their names to less silly ones).
And also added two other list-related use cases to the list (list donors and list donations UCs). These are not included in the component diagram (yet).

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptDec 10 2015, 9:24 AM
gabriel-wmde updated the task description. (Show Details)Jan 4 2016, 3:04 PM
gabriel-wmde updated the task description. (Show Details)Jan 4 2016, 4:00 PM
gabriel-wmde updated the task description. (Show Details)Jan 4 2016, 4:17 PM
gabriel-wmde updated the task description. (Show Details)Jan 4 2016, 5:20 PM
gabriel-wmde updated the task description. (Show Details)Jan 4 2016, 5:47 PM
kai.nissen updated the task description. (Show Details)Jan 4 2016, 5:54 PM
gabriel-wmde updated the task description. (Show Details)Jan 5 2016, 12:33 PM
kai.nissen updated the task description. (Show Details)Jan 5 2016, 12:35 PM
gabriel-wmde updated the task description. (Show Details)Jan 5 2016, 12:45 PM
kai.nissen updated the task description. (Show Details)Jan 5 2016, 12:50 PM
gabriel-wmde updated the task description. (Show Details)Jan 5 2016, 1:00 PM
gabriel-wmde updated the task description. (Show Details)Jan 5 2016, 1:03 PM
kai.nissen updated the task description. (Show Details)Jan 5 2016, 1:41 PM
gabriel-wmde updated the task description. (Show Details)Jan 5 2016, 2:19 PM
kai.nissen updated the task description. (Show Details)Jan 5 2016, 2:58 PM
Restricted Application added a project: WMDE-Fun-Team. · View Herald TranscriptMar 13 2017, 4:02 PM