Opened 2 weeks ago

Last modified 33 hours ago

#16048 assigned Bug/Something is broken

CiviCRM Crashed Our Website

Reported by: Nathan Craig Owned by: Jamie McClelland
Priority: Medium Component: Tech
Keywords: civiCRM Cc:
Sensitive: no

Description

Working with Jamie McClelland who was very kindly guiding me through the process of setting up a detention hotline, I activated CiviCRM's localization feature. Doing this crashed the website. I was making slow but steady progress setting up the hotline workflow, and was showing the demo to one of our members who raised the issue that many people answering the phones are monolingual Spanish speakers. I tried to turn on localizaiton. That led me to the crash.

I just recently upgraded Civi to 5.30 and WP is running 5.5.1 multi-site

To resolve the issue, I tried using the backup in /home/members/avid/sites/chihuahuan.org/backups$ to restore the civicrm database and that didn't fix the issue. Then I tried using the backup for the main WP database to try to restore the website. That didn't work either.

I noticed that the other sites in the multi-site network were working fine. At the network level, I deactivated the CiviCRM plugin which brought the site back up. I'm not sure how to proceed to get Civi back without crashing the website.

What is the next course of action? Having restored the Civi database from before the crash, should I reinstall CiviCRM, like doing an upgrade, to see if that fixes things? Any advice on how to proceed next would be greatly appreciated.

This matter also raises the larger issue that I should have a development site. I understand that these are important to prevent crashes on live sites. Unfortunately, I don't know how to effectively set up a development site for a working WP network. That's probably another ticket, but something I really should address.

Change History (20)

comment:1 Changed 2 weeks ago by Nathan Craig

I tried loading up the most recently backed up civicrm.settings.php and ext and uploading a new copy of civicrm, like doing a backup in reverse to try to restore it. When I went to the update page at the end of the upgrade process, I got the error There has been a critical error on your website. Please check your site admin email inbox for instructions.

I was hoping I'd be able to restore CiviCRM from a backup, but that didn't work. Any ideas? Thank you very much. I had to disable the plugin again to get the WP site running. I have Civi installed in another subdomain and other than a timestamp mismatch it seems to be running fine. The problem, caused by trying to activate localization, seems confined to just the one subdomain's instance.

comment:2 Changed 2 weeks ago by JaimeV

Owner: set to Jamie McClelland
Status: newassigned

I will take a look right now and check if there is any solution that is immediately obvious to me but let's assign this ticket to jamie who is more knowledgeable about CiviCRM.

comment:3 Changed 2 weeks ago by JaimeV

Looking at the error logs for your site is a good starting point to get info about why the site is crashing. Below I am searching for any messages from today related to Civi.

avid-main@daza:~/web$ grep 'Wed Oct 14' /home/members/avid/sites/chihuahuan.org/logs/error.log | grep -i civi | tail -n500
[Wed Oct 14 13:01:53.595877 2020] [proxy_fcgi:error] [pid 4040:tid 139629000890112] [client 73.228.12.173:62217] AH01071: Got error 'PHP message: PHP Fatal error:  Uncaught <table style="border: 1px" cellspacing="0">\n<tr><td colspan="3" style="background: #ff9999"> <b>PEAR_Exception</b>: DB Error: no such field in <b>/home/members/avid/sites/chihuahuan.org/web/wp-content/plugins/civicrm/civicrm/vendor/pear/pear-core-minimal/src/PEAR.php</b> on line <b>922</b></td></tr>\n<tr><td colspan="3" style="background: #ff9999">- <b>DB_Error</b>: DB Error: no such field in <b>unknown</b> on line <b>unknown</b></td></tr>\n<tr><td colspan="3" style="background-color: #aaaaaa; text-align: center; font-weight: bold;">Exception trace</td></tr>\n<tr><td style="text-align: center; background: #cccccc; width:20px; font-weight: bold;">#</td><td style="text-align: center; background: #cccccc; font-weight: bold;">Function</td><td style="text-align: center; background: #cccccc; font-weight: bold;">Location</td></tr>\n<tr><td style="text-align: center;">0</td><td>CRM_Core_Error::exceptionHandler()</td><td>/home/members/avid/sites/chihuahuan.org/web/wp-con...'
[Wed Oct 14 13:04:46.962438 2020] [proxy_fcgi:error] [pid 4040:tid 139629017675520] [client 73.228.12.173:62356] AH01071: Got error 'PHP message: PHP Warning:  Invalid argument supplied for foreach() in /home/members/avid/sites/chihuahuan.org/web/wp-content/plugins/caldera-forms-civicrm-master/includes/class-civicrm-caldera-forms-helper.php on line 514PHP message: PHP Fatal error:  Uncaught CiviCRM_API3_Exception: [undefined: DB Error: no such field\n\n  thrown in /home/members/avid/sites/chihuahuan.org/web/wp-content/plugins/civicrm/civicrm/api/api.php on line 133'
[Wed Oct 14 13:19:00.510397 2020] [proxy_fcgi:error] [pid 20325:tid 139629819967232] [client 73.228.12.173:63138] AH01071: Got error 'PHP message: PHP Fatal error:  Cannot redeclare civi_wp() (previously declared in /home/members/avid/sites/chihuahuan.org/web/wp-content/plugins/civicrm_old/civicrm.php:1983) in /home/members/avid/sites/chihuahuan.org/web/wp-content/plugins/civicrm/civicrm.php on line 1982', referer: https://chihuahuan.org/wp-admin/network/plugins.php?plugin_status=all&paged=1&s
[Wed Oct 14 13:19:02.161058 2020] [proxy_fcgi:error] [pid 4115:tid 139629722334976] [client 73.228.12.173:63143] AH01071: Got error 'PHP message: PHP Fatal error:  Cannot redeclare civi_wp() (previously declared in /home/members/avid/sites/chihuahuan.org/web/wp-content/plugins/civicrm_old/civicrm.php:1983) in /home/members/avid/sites/chihuahuan.org/web/wp-content/plugins/civicrm/civicrm.php on line 1982', referer: https://chihuahuan.org/wp-admin/network/plugins.php?plugin=civicrm%2Fcivicrm.php
[Wed Oct 14 13:19:02.470023 2020] [proxy_fcgi:error] [pid 4040:tid 139629017675520] [client 73.228.12.173:63146] AH01071: Got error 'PHP message: PHP Fatal error:  Cannot redeclare civi_wp() (previously declared in /home/members/avid/sites/chihuahuan.org/web/wp-content/plugins/civicrm_old/civicrm.php:1983) in /home/members/avid/sites/chihuahuan.org/web/wp-content/plugins/civicrm/civicrm.php on line 1982', referer: https://chihuahuan.org/wp-admin/network/plugins.php?plugin=civicrm%2Fcivicrm.php

These may not be related to the problem you describe but are probably worth trying to resolve. At least one of the errors appears to be cuased by leaving the old_civicrm folder in the plugins folder causing code to be read twice.

comment:4 Changed 2 weeks ago by Jamie McClelland

Hi - I've spent some time trying to sort out what is going on.

Can you help orient me? I see that CiviCRM seems to be installed in two different sites - one using the database name main_civi and one with the database name main_civicrm. But I'm not sure which domain names these correspond to. Should it be enabled on both?

I see that with at least one of the CiviCRM databases, it used to have multi-lingual enabled, but then it was turned off without the process completing. If you can help me figure out which database is the right one and which URL it should be point to I think I can force it along.

comment:5 Changed 2 weeks ago by Nathan Craig

civicrm_old I created today when trying to restore the backup. I noticed it produced a second instance in the multi-site plugin list. I didn't want to delete those files if that needed to be preserved for some reason. *I'll zip this, remove it, and store the archive locally until the matter is resolved.*

Where would I look for the website's error logs? Is this something I have access to, if so I would check it periodically.

Yes, CiviCRM is installed on two subdomains in the WP multi-site network. probably didn't name the databases in a very smart way. I didn't really know what I was doing when I set it up. It was, and is, learn as I go. I'm sure I made mistakes, this being one of quite a few. It looks like changing the database name is simply a matter of editing civicrm.settings.php. Perhaps this is some housekeeping I can do down the line to make the setup more understandable.

The primary instance is WP site #4 avid.chihuahuan.org. Its database is man_civicrm.sql. This is the one for which I attempted to enable multi-lingual localization functions.

The second instance is WP site #8 data.chihuahuan.org. Its database is man_civi.sql I don't really use this site that often, it was mostly to try to figure out how to use CiviCRM back when I first installed it. This second instance was running when I last checked but CiviCRM was throwing a date related error.

I can't thank you both enough for looking into this. I really appreciate the top class support, and willingness to help me learn.

comment:6 Changed 2 weeks ago by Nathan Craig

I think I found the logs at /home/members/avid/sites/chihuahuan.org/logs.

civicrm_old zipped, stored locally, and removed from server.

Last edited 2 weeks ago by Nathan Craig (previous) (diff)

comment:7 Changed 2 weeks ago by Jamie McClelland

Perfect - this is exactly what I needed. I'm working on it now (sorry the site is currently down).

comment:8 Changed 2 weeks ago by Jamie McClelland

I think it is working now, with CiviCRM enabled. Can you confirm?

I had to run some code from the command line to de-activate the multi-lingual settings (CRM_Core_I18n_Schema::makeSinglelingual("en_US");). I ran the code using wp's eval-file command, along with --url=avid.chihuhuan.org argument to specify the right WordPress multi site.

However, this wasn't enough, because every time it encountered a failing sql statement it stopped the whole process. So I had to further muck with the CiviCRM code to get it to not stop execution on a failure.

Now, the database should be fully in single language mode and in theory should work - but you may get prompted to run database updates, which you should agree to.

If you hit any other errors - please update the ticket.

comment:9 Changed 2 weeks ago by Nathan Craig

Thank you both the site and CiviCRM appear to be running. I was not prompted for any database updates. I may at some point attempt to implement multi-language support, but will do that on the second instance that is not in the main site.

Last edited 2 weeks ago by Nathan Craig (previous) (diff)

comment:10 Changed 2 weeks ago by Nathan Craig

Resolution: fixed
Status: assignedclosed

comment:11 Changed 13 days ago by Nathan Craig

Resolution: fixed
Status: closedassigned

Multi-language support is going to be really valuable for our needs as much of the work will be in Spanish. I've made good progress setting up the structure of the telephone hotline, but wanted again to try to make this in Spanish. In trying to turn on multi-language support, I'm sorry to say crashed CiviCRM again. Here is what I did this time:

  • backed up all of the sql databases at phpmyadmin. I have all three locally.
  • downloaded the localization files from the CiviCRM website.
  • placed l10n in /home/members/avid/sites/chihuahuan.org/web/wp-content/plugins/civicrm/civicrm
  • placed the contents of sql from the localization tarball into /home/members/avid/sites/chihuahuan.org/web/wp-content/plugins/civicrm/civicrm/sql
  • Went to Administer CiviCRM > Settings > Localization. All of the languages showed up in 'Available Languages` (my thought, great this is going to work).
  • Added in both versions of Spanish. They appeared in the available languages box.
  • Went to the bottom of the page and ticked the box Enable Multiple Languages

poof! I blew it up again. Super sorry. I thought I had taken all of the correct precautions and was doing this properly.

I see Jamie resolved the matter before with (CRM_Core_I18n_Schema::makeSinglelingual("en_US");) using wp's eval-file command, along with --url=avid.chihuhuan.org. I'm afraid I don't know how to do this. I could really use some help getting multi-language support up and running and I'd very much like to understand if this is going to be extra complicated when upgrading versions.

I tried turning off CiviCRM and see that it is not bringing the site back up. I can't get the site running. We have a Governor's panel and some state legislators that will be reviewing some pages on the site related to our anti-detention bill. Those pages need to be running soon, so the status is a bit more pressing. Any help just getting the site running at this point would be much appreciated.

  • I noticed that when I went to the plugin page, CiviCRM was not activated. I tried to activate and deactivate it, but that didn't turn it off.
  • Toggling CiviCRM of and on alters the instance on data.chihuahuan.org but not the main instance on avid.chihuahuan.org. I'm not sure why this is the case.
Last edited 13 days ago by Nathan Craig (previous) (diff)

comment:12 Changed 13 days ago by Nathan Craig

Priority: MediumHigh

comment:13 Changed 12 days ago by Jamie McClelland

My previous tricks are not working - I can't get passed the civicrm error. You said you made a backup of the database. Is it available? Have you tried restoring it?

If you have it and can upload it to your home directory via sftp I can restore it. Or if you can restore it via phpmyadmin that might be the fastest way to get things back up and running.

comment:14 Changed 12 days ago by Nathan Craig

Thanks so much for trying, I know its the weekend so I appreciate it.

Using daza.mayfirst.org/phpmyadmin, I tried importing the file for main_civi.sql.

It returned the following error:

SQL query:

--
-- Functions
--
CREATE DEFINER=`main_admin`@`localhost` FUNCTION `civicrm_strip_non_numeric` (`input` VARCHAR(255)) RETURNS VARCHAR(255) CHARSET utf8 COLLATE utf8_unicode_ci NO SQL
    DETERMINISTIC
BEGIN
      DECLARE output   VARCHAR(255) DEFAULT '';
      DECLARE iterator INT          DEFAULT 1;
      WHILE iterator < (LENGTH(input) + 1) DO
        IF SUBSTRING(input, iterator, 1) IN ('0', '1', '2', '3', '4', '5', '6', '7', '8', '9') THEN
          SET output = CONCAT(output, SUBSTRING(input, iterator, 1));
        END IF;
        SET iterator = iterator + 1;
      END WHILE;
      RETURN output;
    END

MySQL said: Documentation
#1304 - FUNCTION civicrm_strip_non_numeric already exists

Into the following directory, I put all three sql databases I exported:

/home/members/avid/sites/chihuahuan.org/users/avid-main

The one for the crashed site should be main_civi.sql it corresponds to avid.chihuahuan.org which is site 4 in the WP multisite network.

I set CiviCRM to active. If I should set it inactive, let me know and I'll make that change straight away.

Thanks again, so very much, for looking into this.

Last edited 12 days ago by Nathan Craig (previous) (diff)

comment:15 Changed 12 days ago by Jamie McClelland

I just restored the database (I first dropped it and then restored it, which got around the error).

Now it seems that things are working.

We should discuss a bit more the language issues. You don't necessarily need to enable multi-lingual in CiviCRM to get the interface in spanish.

The CiviCRM Multi-lingual option gives you the ability to have translated user content - in other words, every custom field you create can have an english and spanish label. In my experience, this is a lot to add to the user interface and it's not necessary. Also, it really complicates the database - when you enable it, it creates new fields and new tables to accomodate all the data.

However, if you keep things single lingual in CiviCRM, you can dd custom fields withboth the english and spanish in the one label provided (e.g. "First name/Nombre" and "Last Name/Apellido").

I'm not experienced with WordPress. With Drupal, you simply enable the multi-lingual option so you get both spanish and english in WordPress. Then, you configure CiviCRM to inherit the CMS language choice and you are ready to go.

We may need to do more research to see how to do this in Wordpress, but I imagine something similar is possible.

comment:16 Changed 12 days ago by Nathan Craig

Great, I see it is back up. Well done and thanks very much.

I had a sense for but didn't quite understand the difference between adding languages and multi-lingual. The documentation was kind of slim and a little confusing, but you helped clarify.

I'm noticing the following:

  • If I deactivate CiviCRM from the network site it is still running under avid.chihuahuan.org. However, it breaks data.chihuahuan.org.
  • If I activate CiviCRM from the network site it is running under both avid and data.

Something seems to be not synced or the switch isn't quite working. Could this have something to do with the WP database?

Thanks for getting this running now. I'm going to steer clear of multi-lingual for the time being though it is going to be necessary for some of our volunteers. Finding out how to inherit the language from the CMS seems like a suitable workaround. I'll read up and reach out before I take any steps so as to avoid crashing things anew. Thanks again, I REALLY appreciate your help.

Regards, n

comment:17 Changed 7 days ago by Jamie McClelland

Hi Nathan - I'm glad everything is back up and running.

Can you say more about why you have two different CiviCRM instances? Now that there isn't a fire to put out, I would be happy to discuss longer term strategy for managing all of this.

My first suggestion would be to not use WordPress multi site.

May First offers a pretty convenient way to have as many "hosting orders" as you need. The nice thing about separating each WordPress site into a different hosting order is that they can be 100% isolated from each other.

  • If one breaks, it doesn't affect the others
  • if one is compromised it doesn't others
  • if one is difficult to upgrade, you can still upgrade the others

The downside is that you have to maintain three separate wordpress sites, but I think the added overhead is worth the benefits.

The second suggestion is to separate your CiviCRM database (or databases) from your front end web sites. In fact, maybe that is what you are doing with data.chihuahuan.org? This has the same benefits as the ones I listed above - but is particularly useful because sometimes you either really need to upgrade the CiviCRM instance but you don't want to mess with an important front end site. Or, vice versa - you wanted the latest and greatest thing in your front end site, but you don't want to risk breaking CiviCRM.

This separation comes with a bigger cost: you don't get any of the ways CiviCRM can integrate with a front end site - like easily integrating contribution pages or event registration pages. However, most of these benefits are more cosmetic then anything else and again, I think they are outweighed by the benefits.

Curious to get your thoughts.

comment:18 Changed 4 days ago by Nathan Craig

Again, thanks so much for getting the site up and running. I'm very grateful for the support and that you're willing to have this conversation with me. I'm learning a lot through the process.

I had some issues signing back in to the support system, so it took me a moment to respond. I figured out what I needed to do at my end to sign back in. Please forgive my delay.

Here's some context on the questions you ask:

Why WP network

I set up a WP network so that I could spawn issue specific pages quickly and easily while managing them under a single framework. For example, as detention abolitionists we are founding members of efforts to ban private prisons in our state. Using the WP multisite it seemed easy to set up a dedicated site for the bill, and the users/credentials were all the same. At the time, it made sense. Now I'm seeing even that dedicated site I linked above isn't loading properly, and requires cycling all of the plugins for the entire network--only to go down again in a couple of days when WP auto updates.

Why two instances of CiviCRM

As I was using WP multisite, I installed CiviCRM under the network and then I activated the plugin on two different sub-sites. First I did it because some of my early attempts to learn Civi broke the site it was running out of (much like the multi-language flub). I'm not sure how to properly set up development sites, but definitely understanding the importance of doing that. Having two instances of Civi was my effort to learn and develop apart from the main org website. As I make clumsy mistakes, I'd really like to learn best practices for setting up a development site of some kind.

Possible rethinking

At first, WP mult seemed like a good idea. For some things (users are the same across sites and only a single place to maintain updates) its great, but some plugins don't play nice over a multi. Also, as you say--when a part goes down the whole thing can grind to a halt. I don't need to set up small issue dedicated sites that often, so it isn't really that big a deal. I followed the hosting orders and set one up to get started. If everything ran more stable, that would be a fine alternative to what I'm doing now.

Separating CiviCRM

I'm not sure I understand how to separate CiviCRM from a front end site. I thought it had to run through some CMS. I'm guessing you are suggesting running the org front end and having the Civi front end as two different sites. In this way, most visitors won't visit the civi front end. I'd like to be able to track donations and we already use it for registration, but I see the point--if one part breaks, the other part isn't vulnerable.

I suppose we can forward people a meeting registration link, and have them directed to a donation page on a dedicated civiCRM site.

WP Rethink

When I started out, I used wordpress because it is what I knew. At that point, I didn't know about CiviCRM. I've learned it seems to be native with Drupal, and Mayfirst offers one click install of drupal. I'd be willing to learn drupal if it is more solid. I'd like the ease of drupals form frontend. Its kind of a pain to make front end forms for Civi from WP. The native forms are hard to customize, and doing it through Caldera Forms is tedious. Perhaps Backdrop is a sensible choice?

Translation

On the issue of translation, for our purposes we figured out what looks to be a really good and simple solution. Use Chrome and have it translate the page, don't even bother with adding languages. By doing this, all of the interface and all menu drop downs, everything is in Spanish. While there isn't great control over the translation, we can work out the few clunky word translations internally. I wish I'd known that workaround at the start, it seems pretty effective.

Pulling the sites apart isn't something I understand how to do. I can read up on it. If drupal is really superior for Civi, then it probably makes sense to run it under civi. The more I've used WP the less I like it, it seems to be a spam magnet. I've seen some things about migrating WP to drupal. What's your opinion? Thanks again so much for your time.

Last edited 4 days ago by Nathan Craig (previous) (diff)

comment:19 Changed 4 days ago by Nathan Craig

Priority: HighMedium

comment:20 Changed 33 hours ago by Jamie McClelland

Thanks for sharing the big picture! I have a few thoughts in response to what you've said (and some answers), although no silver bullets.

  1. Technically, CiviCRM is not native to Drupal and over the last few years I've seen a lot of WordPress love go into things. However, it's my impression that most of the Drupal core developers develop under Drupal. Also, when it comes to Forms, Drupal has a webform module, which, when combined with the webform_civicrm module provides a very flexible tool for building out forms.

The main problem with Drupal is that you pretty much have to use Drupal 7 right now, and when that version stops being supported (Nov 2022) you'll have to make a giant leap to Drupal 8. OR, as you point out, you can go with Backdrop. It appears that Backdrop is well supported within CiviCRM, but I have little experience with that personally.

So, this is now fully muddled :). I think I'm saying: I don't have a strong opinion on Wordpress vs Drupal vs Backdrop. They each have pro's and con's and ultimately, they should each be able to do the job.

  1. I do think that Wordpress is a good option for your web sites though. It's true that it's a magnet for attacks (because it's so widely used), but Wordpress has a history of being backward compatible and easy to maintain. So, as long as you actually maintain it (by that I mean log in at least once a week and click to run updates every time updates are available), it should work reasonably well as a web site.
  1. Multiple hosting orders. I think dropping wordpress multi-site and moving to multiple hosting orders really is going to be the game changer for you. You can setup as many hosting orders as you need, including multiple dev hosting orders. The nice thing about a hosting order is you can completely delete it and start over at any time.

As a stepping stone, I think first moving CiviCRM to a separate hosting order and perhaps creating a third hosting order as a development/testing site for CiviCRM would be a good move.

With CiviCRM segmented out, you would be in a better position to see if your WordPress multi site makes sense. Maybe without the pressure of upgrading and configuring CiviCRM, the multi-site works?

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.