Joomla Bible Study

...components for the glory of God
Notice
  • The administrator has disabled public write access.
  • Only logged-in/registered users are allowed to contribute to the forum.
Welcome, Guest
Please Login or Register.    Lost Password?

Problems with MP3 files, Downloading and Googlebot
(1 viewing) (1) Guest
Go to bottomPage: 12
TOPIC: Problems with MP3 files, Downloading and Googlebot
#1300
Problems with MP3 files, Downloading and Googlebot 8 Months, 3 Weeks ago  
I converted our Church Website to Joomla a couple of months ago (we're running Joomla 1.5.10) and have used the Bible Study 6.0.11a Component - and the results are great. Very pleased with the layout (although it took a lot of fiddling) and I'me looking forward to 6.1...

However, we've had an issue with Google extracting vast amounts of data from the site during it's indexing passes - so much so that I've had to exclude Google's IP addresses from the site. The site is 84MB - Google has pulled 900MB in one day!

I've tracked the problem to the "Download Icon" that Biblestudy provides, which seems to have TWO problems...

1. I'm asuming that Google is seeing the link, which looks like

.../joomla/index.php/sermons/sermons.html?id=8&view=studieslist&controller=studieslist&task=download

which it does not recognise as a "Media file", and starts to download it - if it had .mp3 on the end, apparently Google will not download it

2. We store our MP3 files on a separate "media" server which does not have any bandwidth restrictions - but our main web server does - and for some reason the MP3 file is downloaded THROUGH our main web server, (hence the large indexing load) which is killing our bandwidth on our web serevr. I've verified this on my local test server, so it's not just an issue with our main internet/ISP server.

The strange thing is that the inbuilt media player goes direct to our media server - but the download icon does not. I would have though that it would be easier to have the download icon point at another server...

I guess what I need is for the "Download Icon" to go directly to our media server - but I can't seem to make it do this.

Am I doing something wrong with the setup - or is this actually a "bug/problem" - if I can't resolve it then I'm going to have to stop using Bible Study as I want Google to index the site, but I can't afford (literally - because I get charged extra if I exceed my bandwidth on our main webserver) the increase in bandwidth...

Hoping that you can help...

Many thanks,

Ian

PS. I've tried "Compatibility" mode and have just tried 6.1 RC4 - exactly the same thing happens.
ha21e
Fresh Boarder
Posts: 3
graphgraph
User Offline Click here to see the profile of this user
The administrator has disabled public write access.
 
#1301
Re:Problems with MP3 files, Downloading and Googlebot 8 Months, 3 Weeks ago  
What is the server/folder path that you have saved for those mp3 files? Is it to your media server?

The download function, class.biblestudydownload.php uses these variables:

$download_file = ''.$server.$path.$filename;

They come from:

$server = $media->spath;
$path = $media->fpath;
$filename = $media->filename;

$media is the db object using id from the mediafile record.

What is your site url?
tmfuller
Administrator
Posts: 855
graph
User Offline Click here to see the profile of this user
"You shall know the truth and the truth shall set you free." JC
The administrator has disabled public write access.
 
#1302
Re:Problems with MP3 files, Downloading and Googlebot 8 Months, 3 Weeks ago  
Many thanks for the quick reply.

We have two servers:

Main web server : www.stpetersyateley.org.uk (this is our main web site)

"media" server : www.stpetersyateley.godsweb.co.uk

You can get to the Sermons List page here : www.stpetersyateley.org.uk/all-sermons?view=studieslist

All the mp3 files are saved to our media server - the settings in each media document in the Bible Study Media entries list are:

server = www.stpetersyateley.godsweb.co.uk
folder = /sermons/
file = Sermon%20yyyymmdd%20hhmm.mp3

I must say that everything seems to work perfectly - the inbuilt media player plays the files, the download icon downloads the files. Searches and categories work in the display of sermons etc. etc.

I can see in the generated HTML that the "object" entry for the media player points (correctly) to the media server. This works fine with data coming direct from the media server to the browser.

However, when you hit the download Icon in the browser, I can see (directly from the network monitor on my test server, and eventually from the hourly stats on our main web server) a pulse of network traffic of exactly the size of the mp3 file - so it looks like the mp3 files is being download through the web server : mp3 file ----> media server -----> web server ----> browser.

Ian.
ha21e
Fresh Boarder
Posts: 3
graphgraph
User Offline Click here to see the profile of this user
The administrator has disabled public write access.
 
#1303
Re:Problems with MP3 files, Downloading and Googlebot 8 Months, 3 Weeks ago  
Yes, Ian - I think you are right. When I click the download icon it appears to come from your web server, not your media server.

Now the question is, why?

There is some code in the download function that may do this - honestly, I am not an expert on this - this code was taken and adapted from docman I think.

$user_agent = (isset($_SERVER["HTTP_USER_AGENT"]) ) ? $_SERVER["HTTP_USER_AGENT"] : $HTTP_USER_AGENT;

I wonder if that changes the user agent to your web server, not your media server.

Basically then this is the download command:

$url = $download_file;
$ch = curl_init();

curl_setopt($ch, CURLOPT_FILE, $out);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_URL, $url);

curl_exec($ch);

We've not run into this problem before - I too have my media files on a different server. The church is at www.CalvaryChapelNewberg.org but the media comes from www.CalvaryChapelNewberg.net which is a GoDaddy server that doesn't have bandwidth problems.

I'm starting to wonder if the same thing happens with my downloads too since the file seems to be coming from the .org site, not the .net site in the download dialogue box.

I think Eugen, one of our developers might be better at answering this question - I believe he will be copied on this thread.

Tom
tmfuller
Administrator
Posts: 855
graph
User Offline Click here to see the profile of this user
"You shall know the truth and the truth shall set you free." JC
The administrator has disabled public write access.
 
#1304
Re:Problems with MP3 files, Downloading and Google 8 Months, 3 Weeks ago  
hmm, well the reason for the download class was to fix a problem with the file download dialog not initiating when linking directly to the mp3 file. However, when we implemented CURL, files are downloaded to the same server (just the way CURL works), and then sent to the user, which is probably why there is excessive bandwidth on your web server, and why files don't seem to be coming from the right server.

Although, with compatibility mode, the mp3 file locations (your media server) are sent to our server here (joomlabiblestudy.org), which we then query your media server to fetch the files, and return them to your webserver -- so, I don't think there shouold be much bandwidth on your webserver because the link is then: User->JoomlaBiblestudy->MediaServer instead of User->Webserver->mediaServer.

I can't be sure of this, however, until I do more research.
genu
Moderator
Posts: 147
graphgraph
User Offline Click here to see the profile of this user
The administrator has disabled public write access.
 
#1791
Re:Problems with MP3 files, Downloading and Google 5 Months, 2 Weeks ago  
genu wrote:
I can't be sure of this, however, until I do more research.
Did you find an answer to this? We'd like to store our media files on a different server, for the same reason, so I'm wondering if we'll hit the same issue.

Thanks.
James Poole
Fresh Boarder
Posts: 2
graphgraph
User Offline Click here to see the profile of this user
The administrator has disabled public write access.
 
Go to topPage: 12
Moderators: Eugen, Darren Vandermyde, Kyle