8bee8e1317
Merge pull request #21317 from FlorentCoppint/s3_upload_part_size
...
Upload part size as S3 parameter instead of constant value
2020-06-09 19:28:58 +02:00
3594ba6971
Upload part size as S3 parameter instead of constant value
...
Some S3 providers need a custom upload part size (500 MB static value in Nextcloud).
Here is a commit to change this value via S3 configuration, instead of using S3_UPLOAD_PART_SIZE constant.
A new parameter is added for an S3 connection : uploadPartSize
Signed-off-by: Florent <florent@coppint.com>
2020-06-09 09:18:42 +02:00
e75797ad67
increase log level for storage self-test
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-06-08 18:12:49 +02:00
6aa6ab3e02
Add lazy events for the Node API
...
Right now if you want to get events via the Node API you have to have a
real instance of the Root. Which in turns sets up the whole FS.
We should make sure this is done lazy. Else enabling the preview
generator for example makes you setup the whole FS on each and every
authenticated call.
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-05-28 12:35:45 +02:00
6b26744787
Clear the statscache before fetching the metadata
...
Else if a lot of writes happen. It might happen that an old stat result
is used. Resulting in a wrong file size for the file. For example the
text app when a lot of people edit at the same time.
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-05-22 09:21:53 +02:00
f7153a3d0c
generalize the "lazy folder" concept
...
makes it easy to use lazy intialized folder in other places
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-05-11 15:36:13 +02:00
d5850eb28f
Merge pull request #18955 from adrb/swift_upload_large_objects
...
Large Object support for OpenStack Swift
2020-05-06 13:03:24 +02:00
4e6d3d1880
Fix getDirectoryContent implementation for Jail wrapper
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-30 15:27:09 +02:00
f6b435d0d9
Run the cs fixer for green ci
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-04-30 10:28:44 +02:00
a1c1b354fc
Merge pull request #18883 from adrb/optimize_swift_14116
...
Optimize Openstack Swift files download
2020-04-30 09:06:19 +02:00
cb057829f7
Update license headers for 19
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-29 11:57:22 +02:00
e107519295
Large Object support for OpenStack Swift.
...
Until now, you wouldn't be able to create
objects larger that 5GB.
It's somewhat related with pull #18883
Signed-off-by: Adrian Brzezinski <adrian.brzezinski@eo.pl>
2020-04-27 14:23:59 +02:00
354c0996d8
It's a fix for issue #14116 ,
...
Improves efficiency when downloading files from Swift storage.
Before, files were downloaded and then pushed back to user.
That behaevior causes all kinds of performance problems.
Now, files are streamed directly to user.
Signed-off-by: Adrian Brzezinski <adrian.brzezinski@eo.pl>
2020-04-27 14:16:50 +02:00
436908ee3f
filter out blacklisted files in getDirectoryContent
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-23 13:34:55 +02:00
9735b5d61f
Add method to storage backends to get directory content with metadata
...
Currently you need to use `opendir` and then call `getMetadata` for
every file, which adds overhead because most storage backends already
get the metadata when doing the `opendir`.
While storagebackends can (and do) use caching to relief this problem,
this adds cache invalidation dificulties and only a limited number of
items are generally cached (to prevent memory usage exploding when
scanning large storages)
With this new methods storage backends can use the child metadata they
got from listing the folder to return metadata without having to keep
seperate caches.
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-20 15:45:04 +02:00
79cf13e349
Merge pull request #16946 from exussum12/patch-1
...
Increase max read
2020-04-16 10:37:10 +02:00
65e134c13a
Update View.php
2020-04-15 19:10:06 +01:00
7fe8477830
Run cs:fix and fix HashWrapper code
...
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2020-04-15 14:08:56 +02:00
d8557d0d8a
Merge pull request #19540 from jlutran/fixKeystoneV3Auth
...
Fix issues with Keystone auth v3 in files_external app
2020-04-15 14:05:32 +02:00
301327135c
Merge pull request #20361 from nextcloud/write-stream-close-on-exception
...
Close the streams in `writeStream` even when there is an exception
2020-04-15 10:48:24 +02:00
3e849daf11
Fix issues with Keystone auth v3 in files_external app
...
Signed-off-by: Julien Lutran <julien.lutran@corp.ovh.com>
2020-04-15 10:33:08 +02:00
c2677682c4
Return hashes of uploaded content for dav uploads
...
hashes are set in "X-Hash-MD5", "X-Hash-SHA1" and "X-Hash-SHA256" headers.
these headers are set for file uploads and the MOVE request at the end of a multipart upload.
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-13 12:57:27 +02:00
75b99e85ff
Increase max read
...
8kb is very low, especially given this will be local files
Signed-off-by: Scott Dutton <scott@exussum.co.uk>
2020-04-12 13:01:29 +01:00
734c62bee0
Format code according to PSR2
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:56:50 +02:00
28f8eb5dba
Add visibility to all constants
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:54:27 +02:00
1584c9ae9c
Add visibility to all methods and position of static keyword
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:51:06 +02:00
a7c8d26d31
Add visibility to all properties and move static keyword
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:48:31 +02:00
caff1023ea
Format control structures, classes, methods and function
...
To continue this formatting madness, here's a tiny patch that adds
unified formatting for control structures like if and loops as well as
classes, their methods and anonymous functions. This basically forces
the constructs to start on the same line. This is not exactly what PSR2
wants, but I think we can have a few exceptions with "our" style. The
starting of braces on the same line is pracrically standard for our
code.
This also removes and empty lines from method/function bodies at the
beginning and end.
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 14:19:56 +02:00
14c996d982
Use elseif instead of else if
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 10:35:09 +02:00
fb64fda4d8
Merge pull request #20385 from nextcloud/skeeable-http-harden
...
harden seekable http stream a bit against failures
2020-04-10 08:34:52 +02:00
44577e4345
Remove trailing and in between spaces
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 16:07:47 +02:00
42625a46be
Remove spaces after method or function call
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 16:05:56 +02:00
60de74ac40
harden seekable http stream a bit against failures
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-09 14:37:01 +02:00
afbd9c4e6e
Unify function spacing to PSR2 recommendation
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 13:54:22 +02:00
41b5e5923a
Use exactly one empty line after the namespace declaration
...
For PSR2
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 11:48:10 +02:00
ccbf3059ba
Close the streams in `writeStream` even when there is an exception
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-08 16:40:56 +02:00
f8a5812f98
Try to use the display name of file transfers
...
Signed-off-by: Joas Schilling <coding@schilljs.com>
2020-04-08 15:41:20 +02:00
f5919d5b83
Merge pull request #20033 from nextcloud/s3-seekable-stream
...
Enable fseek for files in S3 storage
2020-04-06 21:38:49 +02:00
56aa8fd035
Properly hint exceptions thrown by getUserFolder
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2020-04-03 23:06:26 +02:00
e22a28ecc2
update autoloader
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-03 16:27:20 +02:00
3ba46f3b50
add basic tests for s3 seeking and add some error handling if reopen return the wrong range
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 15:23:05 +02:00
7b07e7251c
make seekable s3 stream generic
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 15:21:05 +02:00
14401efb0f
Enable fseek for files in S3 storage
...
Signed-off-by: Lukas Stabe <lukas@stabe.de>
2020-04-01 15:21:01 +02:00
d9184584e0
restart transaction when cache insert fails due to duplicate
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:17:34 +02:00
5199f80196
call cache->insert directly instead of going trough cache->put first when scanning
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:17:28 +02:00
9c7da26394
dont try to get existing cache data during scanning if we already know its a new file
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:17:25 +02:00
3fb32f6f3d
reuse known parentId when inserting cache entries in scanner
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:14:52 +02:00
5439469b68
optimize getMetaData for local storage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:14:48 +02:00
1a9330cd69
Update the license headers for Nextcloud 19
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-31 14:52:54 +02:00
463b388589
Merge pull request #20170 from nextcloud/techdebt/remove-unused-imports
...
Remove unused imports
2020-03-27 17:14:08 +01:00
b80ebc9674
Use the short array syntax, everywhere
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-26 16:34:56 +01:00
74936c49ea
Remove unused imports
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-25 22:08:08 +01:00
23eb0462e5
Use lowercase true, false and null constants
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-25 21:53:04 +01:00
b300dc5a50
Merge pull request #19782 from nextcloud/quota-wrapper-include-external
...
Use global used space in quota wrappen when external storage is included
2020-03-24 21:36:01 +01:00
3443a00621
Actually check if the owner is not null
...
Else this can error out on storages that do not have an owner (e.g.
groupfolders).
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-03-16 13:12:10 +01:00
9a35e789b4
Get correct mimetype on objectstores
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-03-11 11:14:12 +01:00
b6245be302
Merge pull request #19746 from nextcloud/locked-exception-forward-existing
...
pass the existing locks info when making locked exception with absolu…
2020-03-09 19:31:45 +01:00
c32750ddd9
Use global used space in quota wrappen when external storage is included
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-03-04 18:02:50 +01:00
fab22ac14c
pass the existing locks info when making locked exception with absolute paths
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-03-02 19:29:44 +01:00
f972990469
dont try to update storage mtime if we can't get the mtime
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-03-02 15:57:25 +01:00
63608ef461
allow writing content directly when creating new SimpleFile
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-02-28 12:55:22 +01:00
5ca1929e8c
Create SimpleFile only when writing the content
...
instead of first creating an empty file and then writing the content.
This solves the overhead of creating an empty file with the common pattern:
```php
$file = $simpleFilder->newFile('foo.txt');
$file->putContent('bar.txt');
```
roughly halving the number of storage and database operations that need to be done when creating a `SimpleFile`.
This is not automatically done with `File` because that has a more complex api which I'm more hesitant to touch.
Instead the `Folder::newFile` api has been extended to accept the content for the new file.
In my local testing, the overhead of first creating an empty file took about 20% of the time for preview generation
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-02-28 12:55:22 +01:00
dc050fb5e5
Allow to overwrite the path on the cache event
...
Signed-off-by: Joas Schilling <coding@schilljs.com>
2020-02-13 08:38:33 +01:00
08cadf8416
Don't create invalid users
...
Signed-off-by: Joas Schilling <coding@schilljs.com>
2020-02-12 10:55:27 +01:00
1b46621cd3
Update license headers for 18
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2019-12-20 09:23:25 +01:00
b81b824da1
Add typed events for the filesystem/scanner
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2019-12-13 09:41:05 +01:00
c6e51924c8
Merge pull request #18236 from nextcloud/bugfix/noid/always-detect-mimetype-by-content-in-workflows
...
Allow to detect mimetype by content
2019-12-12 11:19:55 +01:00
4356c91ffd
Fix detection of non extention types
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-12-12 09:58:22 +01:00
4fad9e4e6b
Merge pull request #18314 from nextcloud/fixFileIdSearch
...
Fix ambigious fileID on search
2019-12-10 11:28:40 +01:00
107ef72afb
Try more methods
...
Signed-off-by: Joas Schilling <coding@schilljs.com>
2019-12-10 10:56:47 +01:00
41cb214112
fix https://github.com/nextcloud/server/issues/18313
...
Signed-off-by: tobiasKaminsky <tobias@kaminsky.me>
2019-12-10 09:47:30 +01:00
da44c2a414
Some more fixes in detecting the mimetype from the content
...
Signed-off-by: Joas Schilling <coding@schilljs.com>
2019-12-10 09:11:42 +01:00
1336dedd5d
Cleanup the code
...
Signed-off-by: Joas Schilling <coding@schilljs.com>
2019-12-10 09:11:40 +01:00
b92ebb928a
Allow to check for the mimetype by content only
...
Signed-off-by: Joas Schilling <coding@schilljs.com>
2019-12-10 09:10:32 +01:00
5dba8d318d
Fix search limit to home
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-12-09 20:38:13 +01:00
29575c4d36
Move custom definition logic into method
...
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2019-12-09 12:10:19 +01:00
64aba49461
Ensure that we don't merge broken json.
...
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2019-12-09 12:09:43 +01:00
5bf3d1bb38
Update license headers
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2019-12-05 15:38:45 +01:00
63cb31542d
Merge pull request #17941 from nextcloud/search-by-owner
...
Allow filtering the search results to the users home storage
2019-12-05 11:04:33 +01:00
81b919ed9f
Merge pull request #18163 from nextcloud/fix_file_size
...
fix filename size
2019-12-04 14:15:07 +01:00
76b78edd40
Merge pull request #17833 from nextcloud/propagator-no-negative-sizes
...
dont set folder size to negative values during propagation
2019-12-04 11:49:26 +01:00
1110a93fc4
fix filename size, add comment
...
Signed-off-by: Greta Doci <gretadoci@gmail.com>
2019-12-04 11:04:46 +01:00
f9e5b33c2b
remove old 'owner' filter
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-12-03 15:20:54 +01:00
c62637da8b
Allow filtering the search results to the users home storage
...
This is done by adding a
```xml
<d:eq>
<d:prop>
<oc:owner-id/>
</d:prop>
<d:literal>$userId</d:literal>
</d:eq>
```
clause to the search query.
Searching by `owner-id` can only be done with the current user id
and the comparison can not be inside a `<d:not>` or `<d:or>` statement
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-12-03 13:49:37 +01:00
6a940d5c74
Merge pull request #18149 from nextcloud/fix/18148/dont-create-bundle-too-early
...
delay creation of the cert bundle
2019-11-29 09:07:33 +01:00
0e911d643f
delay creation of the cert bundle
...
fixes #18148
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2019-11-28 17:00:55 +01:00
9eb1554fad
Properly anotate LockedException in files node api
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2019-11-28 14:17:15 +01:00
0568b01267
Improve etag handling
...
Check if values exist before using them
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-11-27 13:34:42 +01:00
fc16b09a2c
Files first to use own GenericEvent
...
* those are added to 18 only anyway :)
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2019-11-26 14:48:59 +01:00
74c6beb603
dont set folder size to negative values during propagation
...
normally this shouldn't be a problem, but cache/storage desync might cause this
so this adds some failsafe to ensure we dont corrupt the cache further
the minimum value is set to -1 instead of 0 in order to triger a background scan
on the folder and figure out the size properly
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-11-26 12:05:30 +01:00
c7d5b8fc49
Merge pull request #18079 from nextcloud/fixes/phpcs
...
Some php-cs fixes
2019-11-25 14:07:00 +01:00
68748d4f85
Some php-cs fixes
...
* Order the imports
* No leading slash on imports
* Empty line before namespace
* One line per import
* Empty after imports
* Emmpty line at bottom of file
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-11-22 20:52:10 +01:00
9a64759af5
Harden watcher
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-11-22 13:38:16 +01:00
d625d8bd1e
Merge pull request #17824 from nextcloud/fix/dav/catch_forbidden
...
Catch forbidden http status code
2019-11-20 12:32:00 +01:00
8800a7e890
Fix cache moves
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-11-18 12:48:00 +01:00
c08e803db3
Catch forbidden http status code
...
If you try to do something on a DAV mount (external or federated share)
that is not allowed. We should not mark the storage as not available but
just fail somewhat gracefully.
Now by catching this and just properly returning the operation will just
fail (and notify the user) which is already a lot better then marking
the storage as unavailable and doing boom.
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-11-17 09:11:15 +01:00
f2724992fc
cast ctime and utime
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-11-13 12:39:56 +01:00
4c97181128
filter out empty filecache extension values
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-11-13 12:39:52 +01:00
066faaec05
fix folder delete
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-11-13 12:39:46 +01:00
43fa746ad9
dont create empty filecache extended rows
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-11-13 12:39:43 +01:00
Roeland Jago Douma
Florent
Robin Appelman
Roeland Jago Douma
Christoph Wurst
Adrian Brzezinski
Scott Dutton
Morris Jobke
Christoph Wurst
Julien Lutran
Joas Schilling
Julius Härtl
Lukas Stabe
tobiasKaminsky
Daniel Kesselberg
Greta Doci
Arthur Schiwon